[Qemu-devel] QCow v2

[Qemu-devel] QCow v2

[Nathaniel McCallum]

Mark's notes on the qcow format got me to thinking how useful it would
be to be able to store other information in the qcow image itself.  For
instance you could store the configuration for the virtual machine in
the image which could be extracted and then start the virtual machine.

So... I'm proposing that the qcow image be extended to support this
scenario.  Something as simple as:

  typedef struct QCowHeader {
      uint32_t magic;
      uint32_t version;

      uint8_t  embedded_data_type;
      uint64_t embedded_data_offset;
      uint32_t embedded_data_size;
      uint32_t mtime;

      uint64_t size;

      uint8_t  cluster_bits;
      uint8_t  l2_bits;
      uint32_t crypt_method;

      uint64_t l1_table_offset;
  } QCowHeader;

Thus, embedded_data_type is a constant signifying the type of the data.
embedded_data_type could be a string, etc.  Of course, there are a million
other ways this could be implemented.  The idea being that we could store
more than just backing store.

One possible application could be that you could store config info in
the image and have a qemu-loader app that extracts the info and starts
the virtual machine.  Another application could be just misc. metadata:
copyright, contact info, VM version, etc...

I can provide a patch to block-qcow.c if there is interest.

Nathaniel

Re: [Qemu-devel] QCow v2

[Johannes Schindelin]

Hi,

you said you could come up with a patch easily? Please do; it is much 
easier discussing things with a working prototype.

Also, if you change the version of qcow, please make sure that you keep 
the code backwards compatible: lots of people have qcow images they expect 
to work in 10 years, too.

Ciao,
Dscho

Re: [Qemu-devel] QCow v2

[Raphaël Rigo]

Nathaniel McCallum wrote:
> Mark's notes on the qcow format got me to thinking how useful it would
> be to be able to store other information in the qcow image itself.  For
> instance you could store the configuration for the virtual machine in
> the image which could be extracted and then start the virtual machine.
> 
Hello,
I think it would be much more interesting to store the virtual machine 
in a config file as vmware does. This way you can use a .cow file with 
several virtual machines.

Regards,
Raphaël

Re: [Qemu-devel] QCow v2

[Fabrice Bellard]

Hi,

I am not sure it is interesting to store the configuration of the VM in 
the disk image (there can be several disk images for a given VM). 
Moreover, extensions to the qcow header must be added at its end.

Regarding my current plans, the next qcow evolution will be the support 
for multiple snapshots. Each snapshot will be tagged by a label.

Fabrice.

Nathaniel McCallum wrote:
> Mark's notes on the qcow format got me to thinking how useful it would
> be to be able to store other information in the qcow image itself.  For
> instance you could store the configuration for the virtual machine in
> the image which could be extracted and then start the virtual machine.
> 
> So... I'm proposing that the qcow image be extended to support this
> scenario.  Something as simple as:
> 
>   typedef struct QCowHeader {
>       uint32_t magic;
>       uint32_t version;
> 
>       uint8_t  embedded_data_type;
>       uint64_t embedded_data_offset;
>       uint32_t embedded_data_size;
>       uint32_t mtime;
> 
>       uint64_t size;
> 
>       uint8_t  cluster_bits;
>       uint8_t  l2_bits;
>       uint32_t crypt_method;
> 
>       uint64_t l1_table_offset;
>   } QCowHeader;
> 
> Thus, embedded_data_type is a constant signifying the type of the data.
> embedded_data_type could be a string, etc.  Of course, there are a million
> other ways this could be implemented.  The idea being that we could store
> more than just backing store.
> 
> One possible application could be that you could store config info in
> the image and have a qemu-loader app that extracts the info and starts
> the virtual machine.  Another application could be just misc. metadata:
> copyright, contact info, VM version, etc...
> 
> I can provide a patch to block-qcow.c if there is interest.
> 
> Nathaniel
> 
> 
> 
> _______________________________________________
> Qemu-devel mailing list
> Qemu-devel@nongnu.org
> http://lists.nongnu.org/mailman/listinfo/qemu-devel
> 
> 

Re: [Qemu-devel] QCow v2

[Christian MICHON]

how about also adding lzma compression for qcow ?
I know how to do this inside linux kernel, but lzma'ed qcow images
for windows guests could be a very nice addition too.

Thank Mark for the notes. I'll look at it to see how to start a
lzma patch for qcow...

On 7/4/06, Fabrice Bellard <fabrice@bellard.org> wrote:
> Regarding my current plans, the next qcow evolution will be the support
> for multiple snapshots. Each snapshot will be tagged by a label.
>
> Fabrice.
>

-- 
Christian

Re: [Qemu-devel] QCow v2

[Fabrice Bellard]

If it does not compress significantly better I see no need for that 
(remember that qemu uses 4 KB clusters).

Fabrice.

Christian MICHON wrote:
> how about also adding lzma compression for qcow ?
> I know how to do this inside linux kernel, but lzma'ed qcow images
> for windows guests could be a very nice addition too.
> 
> Thank Mark for the notes. I'll look at it to see how to start a
> lzma patch for qcow...
> 
> On 7/4/06, Fabrice Bellard <fabrice@bellard.org> wrote:
> 
>> Regarding my current plans, the next qcow evolution will be the support
>> for multiple snapshots. Each snapshot will be tagged by a label.
>>
>> Fabrice.
>>
> 

Re: [Qemu-devel] QCow v2

[Julian Seward]

I have used VMware on a Windows host with the vmdk files on a compressed
NTFS partition.  The result of compression was eventually to cause the
vmdk files to become extremely fragmented (tens of thousands of fragments)
which seriously reduces performance in the end.  I'm not sure why this
happens.  So it would be good not to fall into that particular hole.

J


On Tuesday 04 July 2006 11:57, Fabrice Bellard wrote:
> If it does not compress significantly better I see no need for that
> (remember that qemu uses 4 KB clusters).
>
> Fabrice.
>
> Christian MICHON wrote:
> > how about also adding lzma compression for qcow ?
> > I know how to do this inside linux kernel, but lzma'ed qcow images
> > for windows guests could be a very nice addition too.
> >
> > Thank Mark for the notes. I'll look at it to see how to start a
> > lzma patch for qcow...
> >
> > On 7/4/06, Fabrice Bellard <fabrice@bellard.org> wrote:
> >> Regarding my current plans, the next qcow evolution will be the support
> >> for multiple snapshots. Each snapshot will be tagged by a label.
> >>
> >> Fabrice.
>
> _______________________________________________
> Qemu-devel mailing list
> Qemu-devel@nongnu.org
> http://lists.nongnu.org/mailman/listinfo/qemu-devel

Re: [Qemu-devel] QCow v2

[Johannes Schindelin]

Hi,

On Tue, 4 Jul 2006, Christian MICHON wrote:

> how about also adding lzma compression for qcow ?

Why lzma? We already have a dependency on zlib, why not just take that?

Ciao,
Dscho

Re: [Qemu-devel] QCow v2

[Elefterios Stamatogiannakis]

I agree

Lzma and zlib are pretty much equivalent, so there is pretty much 
nothing to be gained except slightly more compression.

On the other hand with lzo (1) there would be quite a considerable speed 
improvement at the cost of compression.

It could also mean that reading on the qcow format could some times be 
faster (with compressible data) than   reading directly from the hard disk.

lefteris

(1) http://www.oberhumer.com/opensource/lzo/

Johannes Schindelin wrote:
> Hi,
> 
> On Tue, 4 Jul 2006, Christian MICHON wrote:
> 
>> how about also adding lzma compression for qcow ?
> 
> Why lzma? We already have a dependency on zlib, why not just take that?
> 
> Ciao,
> Dscho
> 
> 
> _______________________________________________
> Qemu-devel mailing list
> Qemu-devel@nongnu.org
> http://lists.nongnu.org/mailman/listinfo/qemu-devel

Re: [Qemu-devel] QCow v2

[Christian MICHON]

for what it's worth:
qcow with "lzo -9" would become 4155352 bytes (bigger than original
zlib qcow).

yet it's true at the compression and decompression stages that it
feels faster...

On 7/4/06, Elefterios Stamatogiannakis <estama@dblab.ece.ntua.gr> wrote:
> I agree
>
> Lzma and zlib are pretty much equivalent, so there is pretty much
> nothing to be gained except slightly more compression.
>
> On the other hand with lzo (1) there would be quite a considerable speed
> improvement at the cost of compression.
>
> It could also mean that reading on the qcow format could some times be
> faster (with compressible data) than   reading directly from the hard disk.
>
> lefteris
>
> (1) http://www.oberhumer.com/opensource/lzo/
>
> Johannes Schindelin wrote:
> > Hi,
> >
> > On Tue, 4 Jul 2006, Christian MICHON wrote:
> >
> >> how about also adding lzma compression for qcow ?
> >
> > Why lzma? We already have a dependency on zlib, why not just take that?
> >
> > Ciao,
> > Dscho
> >
> >
> > _______________________________________________
> > Qemu-devel mailing list
> > Qemu-devel@nongnu.org
> > http://lists.nongnu.org/mailman/listinfo/qemu-devel
>
>
> _______________________________________________
> Qemu-devel mailing list
> Qemu-devel@nongnu.org
> http://lists.nongnu.org/mailman/listinfo/qemu-devel
>


-- 
Christian

Re: [Qemu-devel] QCow v2

[Natalia Portillo]

Just there is a question:

If improvement is 5%, IS NOT THE SAME with a 5Mb HDD that with a 5Gb  
HDD.

Wanna do testing with more real scenarios?

Like a Windows XP full installation for example?

I'm very sure that there will be a real difference.

Regards

El 04/07/2006, a las 15:51, Christian MICHON escribió:

> for what it's worth:
> qcow with "lzo -9" would become 4155352 bytes (bigger than original
> zlib qcow).
>
> yet it's true at the compression and decompression stages that it
> feels faster...
>
> On 7/4/06, Elefterios Stamatogiannakis <estama@dblab.ece.ntua.gr>  
> wrote:
>> I agree
>>
>> Lzma and zlib are pretty much equivalent, so there is pretty much
>> nothing to be gained except slightly more compression.
>>
>> On the other hand with lzo (1) there would be quite a considerable  
>> speed
>> improvement at the cost of compression.
>>
>> It could also mean that reading on the qcow format could some  
>> times be
>> faster (with compressible data) than   reading directly from the  
>> hard disk.
>>
>> lefteris
>>
>> (1) http://www.oberhumer.com/opensource/lzo/
>>
>> Johannes Schindelin wrote:
>> > Hi,
>> >
>> > On Tue, 4 Jul 2006, Christian MICHON wrote:
>> >
>> >> how about also adding lzma compression for qcow ?
>> >
>> > Why lzma? We already have a dependency on zlib, why not just  
>> take that?
>> >
>> > Ciao,
>> > Dscho
>> >
>> >
>> > _______________________________________________
>> > Qemu-devel mailing list
>> > Qemu-devel@nongnu.org
>> > http://lists.nongnu.org/mailman/listinfo/qemu-devel
>>
>>
>> _______________________________________________
>> Qemu-devel mailing list
>> Qemu-devel@nongnu.org
>> http://lists.nongnu.org/mailman/listinfo/qemu-devel
>>
>
>
> -- 
> Christian
>
>
> _______________________________________________
> Qemu-devel mailing list
> Qemu-devel@nongnu.org
> http://lists.nongnu.org/mailman/listinfo/qemu-devel
>

Re: [Qemu-devel] QCow v2

[Christian MICHON]

ok, I did this experiment (long and painful).

a XP2003/SP1 qemu guest required 964,231,168 bytes qcow image.
zlib qcow image became 459,686,368 bytes.
lzma estimation (4k clusters) is 437,038,838 bytes.

Yes, 5% are still gained, but the time to get the lzma'ed qcow is
disastrous (especially on systems with anti-virus and anti-malware).

Do you still think it's worth it ?

On 7/4/06, Natalia Portillo <claunia@claunia.com> wrote:
> Just there is a question:
>
> If improvement is 5%, IS NOT THE SAME with a 5Mb HDD that with a 5Gb
> HDD.
>
> Wanna do testing with more real scenarios?
>
> Like a Windows XP full installation for example?
>
> I'm very sure that there will be a real difference.
>
> Regards
>
> El 04/07/2006, a las 15:51, Christian MICHON escribió:
>
> > for what it's worth:
> > qcow with "lzo -9" would become 4155352 bytes (bigger than original
> > zlib qcow).
> >
> > yet it's true at the compression and decompression stages that it
> > feels faster...
> >
> > On 7/4/06, Elefterios Stamatogiannakis <estama@dblab.ece.ntua.gr>
> > wrote:
> >> I agree
> >>
> >> Lzma and zlib are pretty much equivalent, so there is pretty much
> >> nothing to be gained except slightly more compression.
> >>
> >> On the other hand with lzo (1) there would be quite a considerable
> >> speed
> >> improvement at the cost of compression.
> >>
> >> It could also mean that reading on the qcow format could some
> >> times be
> >> faster (with compressible data) than   reading directly from the
> >> hard disk.
> >>
> >> lefteris
> >>
> >> (1) http://www.oberhumer.com/opensource/lzo/
> >>
> >> Johannes Schindelin wrote:
> >> > Hi,
> >> >
> >> > On Tue, 4 Jul 2006, Christian MICHON wrote:
> >> >
> >> >> how about also adding lzma compression for qcow ?
> >> >
> >> > Why lzma? We already have a dependency on zlib, why not just
> >> take that?
> >> >
> >> > Ciao,
> >> > Dscho
> >> >
> >> >
> >> > _______________________________________________
> >> > Qemu-devel mailing list
> >> > Qemu-devel@nongnu.org
> >> > http://lists.nongnu.org/mailman/listinfo/qemu-devel
> >>
> >>
> >> _______________________________________________
> >> Qemu-devel mailing list
> >> Qemu-devel@nongnu.org
> >> http://lists.nongnu.org/mailman/listinfo/qemu-devel
> >>
> >
> >
> > --
> > Christian
> >
> >
> > _______________________________________________
> > Qemu-devel mailing list
> > Qemu-devel@nongnu.org
> > http://lists.nongnu.org/mailman/listinfo/qemu-devel
> >
>
>
>
> _______________________________________________
> Qemu-devel mailing list
> Qemu-devel@nongnu.org
> http://lists.nongnu.org/mailman/listinfo/qemu-devel
>


-- 
Christian

Re: [Qemu-devel] QCow v2

[André Braga]

I'm not sure about LZMA, but I'd really like to see this experiment
done with LZO. The compression/decompression speeds of LZO are
fantastic, and I really don't care to shave off every bit of
compressible information as long as it doesn't impact performance as
hard as cqcow does.

There will naturally be people willing to sacrifice speed for space,
so having multiple compression algorithms each targetting a different
scenario would be great.

Something like LZO for compression/decompression speed, LZMA for
maximum speed saving, and deflate (zlib) as a compromise between the
two.

I'm only not sure if 4k is an optimal block size. Given a block size
it's trivial to map which clusters belong to which block, adding that
to a trivial caching algorithm which keeps the "k" latest accessed
blocks in RAM and lazily flushes them to the image (either forced by a
predefined flush frequency of perhaps 2 flushes per minute or by
necessity, to bring another block into the cache; the candidate to go
away could be chosen by LRU).

Just curious: how does CQCOW handle block writes which produce a
compressed cluster larger than the original? Say, the data entropy of
a given cluster increased. Will CQCOW move every block a bit forward
(big, big, big overhead), will it mark the previously occupied space
as free and keep a block index and a bitmap of holes in the file and
then use a best-fit algorithm to choose a block to put in there in the
future (which would mean image-internal fragmentation), or does it
leave the management details to the host file system (and then one has
no control about how fragmented the image gets, unless one runs a
defrag utility on the host)? Is there another solution I'm missing?

Thanks for the attention,
A.



On 7/11/06, Christian MICHON <christian.michon@gmail.com> wrote:
> ok, I did this experiment (long and painful).
>
> a XP2003/SP1 qemu guest required 964,231,168 bytes qcow image.
> zlib qcow image became 459,686,368 bytes.
> lzma estimation (4k clusters) is 437,038,838 bytes.
>
> Yes, 5% are still gained, but the time to get the lzma'ed qcow is
> disastrous (especially on systems with anti-virus and anti-malware).
>
> Do you still think it's worth it ?
>

Re: [Qemu-devel] QCow v2

[Christian MICHON]

1) size... win30.qcow is 7.5Mb (no compress) 3.5Mb (with zlib)
  it should go down between 2.2Mb and 2.5Mb with lzma

2) we can use lzma in a standalone compressor, and the decompressor is
pure C. So at least at the decompressor level, it's not really a dependency...

3) faster decompression...

On 7/4/06, Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:
> Hi,
>
> On Tue, 4 Jul 2006, Christian MICHON wrote:
>
> > how about also adding lzma compression for qcow ?
>
> Why lzma? We already have a dependency on zlib, why not just take that?
>
> Ciao,
> Dscho
>
>
> _______________________________________________
> Qemu-devel mailing list
> Qemu-devel@nongnu.org
> http://lists.nongnu.org/mailman/listinfo/qemu-devel
>


-- 
Christian

Re: [Qemu-devel] QCow v2

[Fabrice Bellard]

Christian MICHON wrote:
> 1) size... win30.qcow is 7.5Mb (no compress) 3.5Mb (with zlib)
>  it should go down between 2.2Mb and 2.5Mb with lzma

Did you compare with 4 KB clusters ? If you compress the whole file it 
says nothing about the compression gain you will have with 4 KB blocks.

Fabrice.

Re: [Qemu-devel] QCow v2

[Christian MICHON]

how do I test it unless I implement it?
Is it enough to split the qcow into the clusters, compress each one of
them, then add the total files sizes ?

how to split into individual clusters a qcow image ?

On 7/4/06, Fabrice Bellard <fabrice@bellard.org> wrote:
> Christian MICHON wrote:
> > 1) size... win30.qcow is 7.5Mb (no compress) 3.5Mb (with zlib)
> >  it should go down between 2.2Mb and 2.5Mb with lzma
>
> Did you compare with 4 KB clusters ? If you compress the whole file it
> says nothing about the compression gain you will have with 4 KB blocks.
>
> Fabrice.
>
>
> _______________________________________________
> Qemu-devel mailing list
> Qemu-devel@nongnu.org
> http://lists.nongnu.org/mailman/listinfo/qemu-devel
>


-- 
Christian

Re: [Qemu-devel] QCow v2

[Fabrice Bellard]

Christian MICHON wrote:
> how do I test it unless I implement it?
> Is it enough to split the qcow into the clusters, compress each one of
> them, then add the total files sizes ?

Yes.

> how to split into individual clusters a qcow image ?

A good approximation is to use an uncompressed qcow image, or even a raw 
image in which you exclude the empty clusters.

Fabrice.

Re: [Qemu-devel] QCow v2

[Christian MICHON]

sorry I was not clear here...

On 7/4/06, Fabrice Bellard <fabrice@bellard.org> wrote:
> > how to split into individual clusters a qcow image ?
>
> A good approximation is to use an uncompressed qcow image, or even a raw
> image in which you exclude the empty clusters.
>
> Fabrice.

how do I cut into the actual 4k clusters an existing qcow image ?
any bash script or executable to perform this trick ?

-- 
Christian

Re: [Qemu-devel] QCow v2

[Christian MICHON]

ok, I used "split -b 4k" on a 7716864 bytes qcow.
I used standard lzma compression and get 1884 7z-clusters
with a grand total of 3558617 bytes (indeed the gain is small).

The standard zlib qcow gave 3704180 bytes...

Does it make sense to have a cluster based compression ?
Indeed the actual compression is not so impressive then (zlib
or not zlib based).

On 7/4/06, Christian MICHON <christian.michon@gmail.com> wrote:
> sorry I was not clear here...
>
> On 7/4/06, Fabrice Bellard <fabrice@bellard.org> wrote:
> > > how to split into individual clusters a qcow image ?
> >
> > A good approximation is to use an uncompressed qcow image, or even a raw
> > image in which you exclude the empty clusters.
> >
> > Fabrice.
>
> how do I cut into the actual 4k clusters an existing qcow image ?
> any bash script or executable to perform this trick ?
>
> --
> Christian
>


-- 
Christian

Re: [Qemu-devel] QCow v2

[Johannes Schindelin]

Hi,

On Tue, 4 Jul 2006, Christian MICHON wrote:

> ok, I used "split -b 4k" on a 7716864 bytes qcow.
> I used standard lzma compression and get 1884 7z-clusters
> with a grand total of 3558617 bytes (indeed the gain is small).
> 
> The standard zlib qcow gave 3704180 bytes...

That is what I expected.

> Does it make sense to have a cluster based compression ?
> Indeed the actual compression is not so impressive then (zlib
> or not zlib based).

Absolutely! Remember, you want to emulate a _random_ access device here. 
And it is no good if you need 10 minutes just to access the last cluster 
of the emulated hard drive, especially if this is the only one you are 
interested in.

Hth,
Dscho

Re: [Qemu-devel] QCow v2

[Nathaniel McCallum]

On Tue, 2006-07-04 at 11:23 +0200, Fabrice Bellard wrote:
> Hi,
> 
> I am not sure it is interesting to store the configuration of the VM in 
> the disk image (there can be several disk images for a given VM). 
> Moreover, extensions to the qcow header must be added at its end.

Storing the config in the disk image does not preclude:
  1. using multiple disk images with a vm
  2. using multiple vm's with a single disk image

Nor does it make it more difficult to do any of the above.  All it does
is make it easier for the use case of one VM with one image, which is
the vast majority of VM's out there.

In addition, it definitely is interesting to store other random data in
the image.  Here is a second proposal:

We have a key/value section that replaces the backing store section:

  typedef struct QCowHeader {
      uint32_t magic;
      uint32_t version;

      uint64_t key_val_offset; /* was backing_file_offset */
      uint32_t key_val_size;   /* was backing_file_size */
      uint32_t mtime;

      uint64_t size;

      uint8_t  cluster_bits;
      uint8_t  l2_bits;
      uint32_t crypt_method;

      uint64_t l1_table_offset;
  } QCowHeader;

The above does not change the size of the header.  And as such, you can
still read the header off the file and determine how to interpret it 
based on the version number.

The key_val section is pairs of null terminated (UTF-8?) strings.  The
first string in the pair is the key, the second is the value.

It could handle the current backing store design like this:
  "backing_store": "<filename>"

> Regarding my current plans, the next qcow evolution will be the support 
> for multiple snapshots. Each snapshot will be tagged by a label.

I assume you'll do this with multiple backing stores:
  "num_backing_stores"   : "2"
  "backing_store_0"      : "<filename>"
  "backing_store_0_label": "<label>"
  "backing_store_1"      : "<filename>"
  "backing_store_1_label": "<label>"