Re: [Qemu-devel] QCOW2 support for LZO compression

["Richard W.M. Jones" <rjones@redhat.com>]

On Mon, Jun 26, 2017 at 12:12:05PM +0100, Daniel P. Berrange wrote:
> On Mon, Jun 26, 2017 at 10:28:38AM +0200, Kevin Wolf wrote:
> > [ Cc: qemu-devel; don't post to qemu-block only! ]
> > 
> > Am 26.06.2017 um 09:57 hat Peter Lieven geschrieben:
> > > Hi,
> > > 
> > > I am currently working on optimizing speed for compressed QCOW2
> > > images. We use them for templates and would also like to use them for
> > > backups, but the latter is almost infeasible because using gzip for
> > > compression is horribly slow. I tried to experiment with different
> > > options to deflate, but in the end I think its better to use a
> > > different compression algorithm for cases where speed matters. As we
> > > already have probing for it in configure and as it is widely used I
> > > would like to use LZO for that purpose. I think it would be best to
> > > have a flag to indicate that compressed blocks use LZO compression,
> > > but I would need a little explaination which of the feature fields I
> > > have to use to prevent an older (incompatible) Qemu opening LZO
> > > compressed QCOW2 images.
> > > 
> > > I also have already some numbers. I converted a fresh Debian 9 Install
> > > which has an uncomressed QCOW2 size of 1158 MB with qemu-img to a
> > > compressed QCOW2.  With GZIP compression the result is 356MB whereas
> > > the LZO version is 452MB. However, the current GZIP variant uses 35
> > > seconds for this operation where LZO only needs 4 seconds. I think is
> > > is a good trade in especially when its optional so the user can
> > > choose.
> > > 
> > > What are your thoughts?
> > 
> > We had a related RFC patch by Den earlier this year, which never
> > received many comment and never got out of RFC:
> > 
> > https://lists.gnu.org/archive/html/qemu-devel/2017-03/msg04682.html
> > 
> > So he chose a different algorithm (zstd). When I asked, he posted a
> > comparison of algorithms (however a generic one and not measured in the
> > context of qemu) that suggests that LZO would be slightly faster, but
> > have a considerable worse compression ratio with the settings that were
> > benchmarked.
> > 
> > I think it's clear that if there is any serious interest in compression,
> > we'll want to support at least one more algorithm. What we still need to
> > evaluate is which one(s) to take, and whether a simple incompatible flag
> > in the header like in Den's patch is enough or whether we should add a
> > whole new header field for the compression algorithm (like we already
> > have for encryption).
> 
> We might also want to consider whether doing compression on individual
> qcow2 clusters is the best approach for desired usage scenarios, as
> compared to adding a general purpose compression block driver in between
> the qcow2 driver and the file driver. eg akin to just running 'gzip' over
> the entire qcow2 file. The relatively small size of qcow2 clusters limits
> the effectiveness of all compression algorithms we might choose between.
> By adding a separate compression driver below qcow2, we can choose a much
> larger compression block size, independant of qcow2 cluster size.
> 
> Taking a random disk image I have with 1 GB of data. If I tell 'xz' to
> compress using 65 KB block sizes (to simulate compression attained if
> compressing individual qcow2 clusters in isolation), then I get a file
> 124 MB in size, which is barely better than that attained with qcow2's
> built-in gzip compression. If I tell 'xz' to compress with 16 MB block
> sizes though, the output is 114 MB in size, which is a massive win over
> gzip.

I think we need both.  qcow2 block compression allows for writable
qcow2 files, whereas xz compression over a whole file is read-only.
Both have real use cases.

liblzma (xz) is a very flexible library which allows for (read-only)
decompression of individual blocks.  If you prepare your xz file with
the --blocksize parameter, keeping it to something small like 16 MB,
then you can make random-access xz-compressed disk images quite
easily.  In fact I wrote code to do exactly that:

https://github.com/libguestfs/nbdkit/tree/master/plugins/xz

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-top is 'top' for virtual machines.  Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://people.redhat.com/~rjones/virt-top