Re: [Qemu-devel] QCOW2 support for LZO compression

["Denis V. Lunev" <den@openvz.org>]

On 06/26/2017 01:04 PM, Daniel P. Berrange wrote:
> On Mon, Jun 26, 2017 at 11:20:33AM +0200, Peter Lieven wrote:
>> Am 26.06.2017 um 10:28 schrieb Kevin Wolf:
>>> [ Cc: qemu-devel; don't post to qemu-block only! ]
>>>
>>> Am 26.06.2017 um 09:57 hat Peter Lieven geschrieben:
>>>> Hi,
>>>>
>>>> I am currently working on optimizing speed for compressed QCOW2
>>>> images. We use them for templates and would also like to use them for
>>>> backups, but the latter is almost infeasible because using gzip for
>>>> compression is horribly slow. I tried to experiment with different
>>>> options to deflate, but in the end I think its better to use a
>>>> different compression algorithm for cases where speed matters. As we
>>>> already have probing for it in configure and as it is widely used I
>>>> would like to use LZO for that purpose. I think it would be best to
>>>> have a flag to indicate that compressed blocks use LZO compression,
>>>> but I would need a little explaination which of the feature fields I
>>>> have to use to prevent an older (incompatible) Qemu opening LZO
>>>> compressed QCOW2 images.
>>>>
>>>> I also have already some numbers. I converted a fresh Debian 9 Install
>>>> which has an uncomressed QCOW2 size of 1158 MB with qemu-img to a
>>>> compressed QCOW2.  With GZIP compression the result is 356MB whereas
>>>> the LZO version is 452MB. However, the current GZIP variant uses 35
>>>> seconds for this operation where LZO only needs 4 seconds. I think is
>>>> is a good trade in especially when its optional so the user can
>>>> choose.
>>>>
>>>> What are your thoughts?
>>> We had a related RFC patch by Den earlier this year, which never
>>> received many comment and never got out of RFC:
>>>
>>> https://lists.gnu.org/archive/html/qemu-devel/2017-03/msg04682.html
>> I was not aware of that one. Thanks for pointing out.
>>
>>> So he chose a different algorithm (zstd). When I asked, he posted a
>>> comparison of algorithms (however a generic one and not measured in the
>>> context of qemu) that suggests that LZO would be slightly faster, but
>>> have a considerable worse compression ratio with the settings that were
>>> benchmarked.
>> My idea to choose LZO was that it is widely available and available in
>> any distro you can think of. We already have probing for it in configure.
>> My concern with ZSTD would be that it seems there are no packages
>> available for most distros and that it seems to be multi-threaded. I don't
>> know if this will cause any trouble?
> As a distro maintainer I'd always prefer option to use a library that is
> already widely available. While ZSTD could certainly be added to distros,
> it means the QEMU maintainer will end up having to package it & become
> the defacto long term maintainer of it long term, which is an extra burden.
>
> WRT to making compression algorithms configurable, I think it is important
> to ensure we don't add lots of optional algorithms. An important factor is
> portability of images - we don't want to end up with each distro's build
> of QEMU enabling a different sub-set of compression algorithms, as that is
> going to harm interoperability for distributed images. This again makes me
> prefer a compression format whose library is widely available, as that makes
> it highly likely that the distro will choose to enable it during build.
>
> Regards,
> Daniel
Speaking about image compression - ZSDT could be questioned
for different things. Though speaking about online migration -
ZSTD IMHO is the best choice. I have started the investigation
keeping that in mind.

Den