Re: [Qemu-devel] QCOW2 support for LZO compression

Re: [Qemu-devel] QCOW2 support for LZO compression

[Kevin Wolf]

[ Cc: qemu-devel; don't post to qemu-block only! ]

Am 26.06.2017 um 09:57 hat Peter Lieven geschrieben:
> Hi,
> 
> I am currently working on optimizing speed for compressed QCOW2
> images. We use them for templates and would also like to use them for
> backups, but the latter is almost infeasible because using gzip for
> compression is horribly slow. I tried to experiment with different
> options to deflate, but in the end I think its better to use a
> different compression algorithm for cases where speed matters. As we
> already have probing for it in configure and as it is widely used I
> would like to use LZO for that purpose. I think it would be best to
> have a flag to indicate that compressed blocks use LZO compression,
> but I would need a little explaination which of the feature fields I
> have to use to prevent an older (incompatible) Qemu opening LZO
> compressed QCOW2 images.
> 
> I also have already some numbers. I converted a fresh Debian 9 Install
> which has an uncomressed QCOW2 size of 1158 MB with qemu-img to a
> compressed QCOW2.  With GZIP compression the result is 356MB whereas
> the LZO version is 452MB. However, the current GZIP variant uses 35
> seconds for this operation where LZO only needs 4 seconds. I think is
> is a good trade in especially when its optional so the user can
> choose.
> 
> What are your thoughts?

We had a related RFC patch by Den earlier this year, which never
received many comment and never got out of RFC:

https://lists.gnu.org/archive/html/qemu-devel/2017-03/msg04682.html

So he chose a different algorithm (zstd). When I asked, he posted a
comparison of algorithms (however a generic one and not measured in the
context of qemu) that suggests that LZO would be slightly faster, but
have a considerable worse compression ratio with the settings that were
benchmarked.

I think it's clear that if there is any serious interest in compression,
we'll want to support at least one more algorithm. What we still need to
evaluate is which one(s) to take, and whether a simple incompatible flag
in the header like in Den's patch is enough or whether we should add a
whole new header field for the compression algorithm (like we already
have for encryption).

Kevin

Re: [Qemu-devel] QCOW2 support for LZO compression

[Peter Lieven]

Am 26.06.2017 um 10:28 schrieb Kevin Wolf:
> [ Cc: qemu-devel; don't post to qemu-block only! ]
>
> Am 26.06.2017 um 09:57 hat Peter Lieven geschrieben:
>> Hi,
>>
>> I am currently working on optimizing speed for compressed QCOW2
>> images. We use them for templates and would also like to use them for
>> backups, but the latter is almost infeasible because using gzip for
>> compression is horribly slow. I tried to experiment with different
>> options to deflate, but in the end I think its better to use a
>> different compression algorithm for cases where speed matters. As we
>> already have probing for it in configure and as it is widely used I
>> would like to use LZO for that purpose. I think it would be best to
>> have a flag to indicate that compressed blocks use LZO compression,
>> but I would need a little explaination which of the feature fields I
>> have to use to prevent an older (incompatible) Qemu opening LZO
>> compressed QCOW2 images.
>>
>> I also have already some numbers. I converted a fresh Debian 9 Install
>> which has an uncomressed QCOW2 size of 1158 MB with qemu-img to a
>> compressed QCOW2.  With GZIP compression the result is 356MB whereas
>> the LZO version is 452MB. However, the current GZIP variant uses 35
>> seconds for this operation where LZO only needs 4 seconds. I think is
>> is a good trade in especially when its optional so the user can
>> choose.
>>
>> What are your thoughts?
> We had a related RFC patch by Den earlier this year, which never
> received many comment and never got out of RFC:
>
> https://lists.gnu.org/archive/html/qemu-devel/2017-03/msg04682.html

I was not aware of that one. Thanks for pointing out.

>
> So he chose a different algorithm (zstd). When I asked, he posted a
> comparison of algorithms (however a generic one and not measured in the
> context of qemu) that suggests that LZO would be slightly faster, but
> have a considerable worse compression ratio with the settings that were
> benchmarked.

My idea to choose LZO was that it is widely available and available in
any distro you can think of. We already have probing for it in configure.
My concern with ZSTD would be that it seems there are no packages
available for most distros and that it seems to be multi-threaded. I don't
know if this will cause any trouble?

>
> I think it's clear that if there is any serious interest in compression,
> we'll want to support at least one more algorithm. What we still need to
> evaluate is which one(s) to take, and whether a simple incompatible flag
> in the header like in Den's patch is enough or whether we should add a
> whole new header field for the compression algorithm (like we already
> have for encryption).

 From my side there clearly is interest in optimizing the compression. Its
even possible to speed up zlib by 3-4x times by choosing other parameters
for deflate which unfortunately are not compatible with our inflate settings.

I don't know if its worth creating a new header field. Even if we spent to bits
in the end (one for LZO and one for ZSDT). I think this wouldn't hurt. However,
there are likely to pop up new compression algorithms in the future and
a header would be more flexible.

I just don't want to make it too complicated and as you pointed out compression is
not that interesting for most people - maybe due to its speed.

Peter

Re: [Qemu-devel] QCOW2 support for LZO compression

[Denis V. Lunev]

On 06/26/2017 12:20 PM, Peter Lieven wrote:
>
> Am 26.06.2017 um 10:28 schrieb Kevin Wolf:
>> [ Cc: qemu-devel; don't post to qemu-block only! ]
>>
>> Am 26.06.2017 um 09:57 hat Peter Lieven geschrieben:
>>> Hi,
>>>
>>> I am currently working on optimizing speed for compressed QCOW2
>>> images. We use them for templates and would also like to use them for
>>> backups, but the latter is almost infeasible because using gzip for
>>> compression is horribly slow. I tried to experiment with different
>>> options to deflate, but in the end I think its better to use a
>>> different compression algorithm for cases where speed matters. As we
>>> already have probing for it in configure and as it is widely used I
>>> would like to use LZO for that purpose. I think it would be best to
>>> have a flag to indicate that compressed blocks use LZO compression,
>>> but I would need a little explaination which of the feature fields I
>>> have to use to prevent an older (incompatible) Qemu opening LZO
>>> compressed QCOW2 images.
>>>
>>> I also have already some numbers. I converted a fresh Debian 9 Install
>>> which has an uncomressed QCOW2 size of 1158 MB with qemu-img to a
>>> compressed QCOW2.  With GZIP compression the result is 356MB whereas
>>> the LZO version is 452MB. However, the current GZIP variant uses 35
>>> seconds for this operation where LZO only needs 4 seconds. I think is
>>> is a good trade in especially when its optional so the user can
>>> choose.
>>>
>>> What are your thoughts?
>> We had a related RFC patch by Den earlier this year, which never
>> received many comment and never got out of RFC:
>>
>> https://lists.gnu.org/archive/html/qemu-devel/2017-03/msg04682.html
>
> I was not aware of that one. Thanks for pointing out.
>
>>
>> So he chose a different algorithm (zstd). When I asked, he posted a
>> comparison of algorithms (however a generic one and not measured in the
>> context of qemu) that suggests that LZO would be slightly faster, but
>> have a considerable worse compression ratio with the settings that were
>> benchmarked.
>
> My idea to choose LZO was that it is widely available and available in
> any distro you can think of. We already have probing for it in configure.
> My concern with ZSTD would be that it seems there are no packages
> available for most distros and that it seems to be multi-threaded. I
> don't
> know if this will cause any trouble?
>
We have had that compression working in multithreaded process.

>>
>> I think it's clear that if there is any serious interest in compression,
>> we'll want to support at least one more algorithm. What we still need to
>> evaluate is which one(s) to take, and whether a simple incompatible flag
>> in the header like in Den's patch is enough or whether we should add a
>> whole new header field for the compression algorithm (like we already
>> have for encryption).
>
> From my side there clearly is interest in optimizing the compression. Its
> even possible to speed up zlib by 3-4x times by choosing other parameters
> for deflate which unfortunately are not compatible with our inflate
> settings.
>
> I don't know if its worth creating a new header field. Even if we
> spent to bits
> in the end (one for LZO and one for ZSDT). I think this wouldn't hurt.
> However,
> there are likely to pop up new compression algorithms in the future and
> a header would be more flexible.
>
> I just don't want to make it too complicated and as you pointed out
> compression is
> not that interesting for most people - maybe due to its speed.
>
I think we need something generic but simple. I think that we should not
support compression with the different algorithm in the single file.

Speaking about compression, we do have different constraints for
different situation, f.e. backups are written once and rarely read while
generic compression in backing store is read frequently but never
read. Thus the exact algorithm should be selectable.

Den

Re: [Qemu-devel] QCOW2 support for LZO compression

[Peter Lieven]

Am 26.06.2017 um 11:33 schrieb Denis V. Lunev:
> On 06/26/2017 12:20 PM, Peter Lieven wrote:
>> Am 26.06.2017 um 10:28 schrieb Kevin Wolf:
>>> [ Cc: qemu-devel; don't post to qemu-block only! ]
>>>
>>> Am 26.06.2017 um 09:57 hat Peter Lieven geschrieben:
>>>> Hi,
>>>>
>>>> I am currently working on optimizing speed for compressed QCOW2
>>>> images. We use them for templates and would also like to use them for
>>>> backups, but the latter is almost infeasible because using gzip for
>>>> compression is horribly slow. I tried to experiment with different
>>>> options to deflate, but in the end I think its better to use a
>>>> different compression algorithm for cases where speed matters. As we
>>>> already have probing for it in configure and as it is widely used I
>>>> would like to use LZO for that purpose. I think it would be best to
>>>> have a flag to indicate that compressed blocks use LZO compression,
>>>> but I would need a little explaination which of the feature fields I
>>>> have to use to prevent an older (incompatible) Qemu opening LZO
>>>> compressed QCOW2 images.
>>>>
>>>> I also have already some numbers. I converted a fresh Debian 9 Install
>>>> which has an uncomressed QCOW2 size of 1158 MB with qemu-img to a
>>>> compressed QCOW2.  With GZIP compression the result is 356MB whereas
>>>> the LZO version is 452MB. However, the current GZIP variant uses 35
>>>> seconds for this operation where LZO only needs 4 seconds. I think is
>>>> is a good trade in especially when its optional so the user can
>>>> choose.
>>>>
>>>> What are your thoughts?
>>> We had a related RFC patch by Den earlier this year, which never
>>> received many comment and never got out of RFC:
>>>
>>> https://lists.gnu.org/archive/html/qemu-devel/2017-03/msg04682.html
>> I was not aware of that one. Thanks for pointing out.
>>
>>> So he chose a different algorithm (zstd). When I asked, he posted a
>>> comparison of algorithms (however a generic one and not measured in the
>>> context of qemu) that suggests that LZO would be slightly faster, but
>>> have a considerable worse compression ratio with the settings that were
>>> benchmarked.
>> My idea to choose LZO was that it is widely available and available in
>> any distro you can think of. We already have probing for it in configure.
>> My concern with ZSTD would be that it seems there are no packages
>> available for most distros and that it seems to be multi-threaded. I
>> don't
>> know if this will cause any trouble?
>>
> We have had that compression working in multithreaded process.

Do you have a patchset ready ?

>
>>> I think it's clear that if there is any serious interest in compression,
>>> we'll want to support at least one more algorithm. What we still need to
>>> evaluate is which one(s) to take, and whether a simple incompatible flag
>>> in the header like in Den's patch is enough or whether we should add a
>>> whole new header field for the compression algorithm (like we already
>>> have for encryption).
>>  From my side there clearly is interest in optimizing the compression. Its
>> even possible to speed up zlib by 3-4x times by choosing other parameters
>> for deflate which unfortunately are not compatible with our inflate
>> settings.
>>
>> I don't know if its worth creating a new header field. Even if we
>> spent to bits
>> in the end (one for LZO and one for ZSDT). I think this wouldn't hurt.
>> However,
>> there are likely to pop up new compression algorithms in the future and
>> a header would be more flexible.
>>
>> I just don't want to make it too complicated and as you pointed out
>> compression is
>> not that interesting for most people - maybe due to its speed.
>>
> I think we need something generic but simple. I think that we should not
> support compression with the different algorithm in the single file.

Agreed. The compression algorithm should be specified at create time.

>
> Speaking about compression, we do have different constraints for
> different situation, f.e. backups are written once and rarely read while
> generic compression in backing store is read frequently but never
> read. Thus the exact algorithm should be selectable.

Okay, so we have gzip (default) and zstd. If the patches for ZStd are ready
I don't mind to drop support for LZO. I think both ZStd and LZO both offer
good compression and decompression speed compared to gzip.

Peter

Re: [Qemu-devel] QCOW2 support for LZO compression

[Laszlo Ersek]

On 06/26/17 11:33, Denis V. Lunev wrote:
> On 06/26/2017 12:20 PM, Peter Lieven wrote:
>>
>> Am 26.06.2017 um 10:28 schrieb Kevin Wolf:
>>> [ Cc: qemu-devel; don't post to qemu-block only! ]
>>>
>>> Am 26.06.2017 um 09:57 hat Peter Lieven geschrieben:
>>>> Hi,
>>>>
>>>> I am currently working on optimizing speed for compressed QCOW2
>>>> images. We use them for templates and would also like to use them for
>>>> backups, but the latter is almost infeasible because using gzip for
>>>> compression is horribly slow. I tried to experiment with different
>>>> options to deflate, but in the end I think its better to use a
>>>> different compression algorithm for cases where speed matters. As we
>>>> already have probing for it in configure and as it is widely used I
>>>> would like to use LZO for that purpose. I think it would be best to
>>>> have a flag to indicate that compressed blocks use LZO compression,
>>>> but I would need a little explaination which of the feature fields I
>>>> have to use to prevent an older (incompatible) Qemu opening LZO
>>>> compressed QCOW2 images.
>>>>
>>>> I also have already some numbers. I converted a fresh Debian 9 Install
>>>> which has an uncomressed QCOW2 size of 1158 MB with qemu-img to a
>>>> compressed QCOW2.  With GZIP compression the result is 356MB whereas
>>>> the LZO version is 452MB. However, the current GZIP variant uses 35
>>>> seconds for this operation where LZO only needs 4 seconds. I think is
>>>> is a good trade in especially when its optional so the user can
>>>> choose.
>>>>
>>>> What are your thoughts?
>>> We had a related RFC patch by Den earlier this year, which never
>>> received many comment and never got out of RFC:
>>>
>>> https://lists.gnu.org/archive/html/qemu-devel/2017-03/msg04682.html
>>
>> I was not aware of that one. Thanks for pointing out.
>>
>>>
>>> So he chose a different algorithm (zstd). When I asked, he posted a
>>> comparison of algorithms (however a generic one and not measured in the
>>> context of qemu) that suggests that LZO would be slightly faster, but
>>> have a considerable worse compression ratio with the settings that were
>>> benchmarked.
>>
>> My idea to choose LZO was that it is widely available and available in
>> any distro you can think of. We already have probing for it in configure.
>> My concern with ZSTD would be that it seems there are no packages
>> available for most distros and that it seems to be multi-threaded. I
>> don't
>> know if this will cause any trouble?
>>
> We have had that compression working in multithreaded process.
> 
>>>
>>> I think it's clear that if there is any serious interest in compression,
>>> we'll want to support at least one more algorithm. What we still need to
>>> evaluate is which one(s) to take, and whether a simple incompatible flag
>>> in the header like in Den's patch is enough or whether we should add a
>>> whole new header field for the compression algorithm (like we already
>>> have for encryption).
>>
>> From my side there clearly is interest in optimizing the compression. Its
>> even possible to speed up zlib by 3-4x times by choosing other parameters
>> for deflate which unfortunately are not compatible with our inflate
>> settings.
>>
>> I don't know if its worth creating a new header field. Even if we
>> spent to bits
>> in the end (one for LZO and one for ZSDT). I think this wouldn't hurt.
>> However,
>> there are likely to pop up new compression algorithms in the future and
>> a header would be more flexible.
>>
>> I just don't want to make it too complicated and as you pointed out
>> compression is
>> not that interesting for most people - maybe due to its speed.
>>
> I think we need something generic but simple. I think that we should not
> support compression with the different algorithm in the single file.
> 
> Speaking about compression, we do have different constraints for
> different situation, f.e. backups are written once and rarely read while
> generic compression in backing store is read frequently but never
> read. Thus the exact algorithm should be selectable.

Pluggable / selectable compression methods are likely the most flexible
and future-proof. A new header sounds good to me (... said by someone
who comments on this from the sidelines.)

I would advise caution against multi-threaded compression libraries.
Unless they are coded very-very carefully with regard to signal handling
and general error handling / propagation, they cannot be considered
"opaque" enough.

(I had written and maintained the original (0.x) branch of "lbzip2",
which was extremely conscious of error handling and signals. That was a
challenge even in a standalone program, and I didn't even attempt to
retrofit the code to the existing libbz2 APIs (i.e. I never even tried
to librarize the code).)

This does not mean that people cannot get such a library right. It's
just that *by default* such a library will have a number of obscure bugs
related to: signals, forking, and general error handling. It could also
have problems with unbounded memory allocation. An MT compression
library that gets all of this right is the exception IMO, not the norm.
(I don't know anything about the ZSTD library; it could be such a high
quality library.)

Another complication with MT *de*compression is that the CPU demand from
the IO thread (which is by default responsible for handling IO, when not
using dataplane -- is that right?) would "leak" to other physical
processors. I believe this can interfere with use cases where people
carefully isolate host CPUs between "QEMU" and "non-QEMU" workloads,
plus pin QEMU's VCPU threads, and IO threads, to different host CPUs
(see vcpupin / emulatorpin / iothreadpin under
<http://libvirt.org/formatdomain.html#elementsCPUTuning>.) It's probably
possible to figure out the right thing for "ZSTD threads" as well, but
IMO it remains a complication nonetheless.

Personally I would recommend a new header, and LZO, as a starting point.
As pointed out above, LZO is widely available in distros. It has good
performance, and it is single-threaded similarly to zlib. I use LZO for
two QEMU-related purposes ATM:

- I use the kdump-lzo format when dumping guest memory
  (virsh dump $DOMAIN $CORE_FLE --memory-only --format kdump-lzo)

- I use LZO compression for "virsh managedsave"
  (by setting "save_image_format" in "/etc/libvirt/qemu.conf" to "lzop")

Thanks
Laszlo

Re: [Qemu-devel] QCOW2 support for LZO compression

[Denis V. Lunev]

On 06/26/2017 01:16 PM, Laszlo Ersek wrote:
> On 06/26/17 11:33, Denis V. Lunev wrote:
>> On 06/26/2017 12:20 PM, Peter Lieven wrote:
>>> Am 26.06.2017 um 10:28 schrieb Kevin Wolf:
>>>> [ Cc: qemu-devel; don't post to qemu-block only! ]
>>>>
>>>> Am 26.06.2017 um 09:57 hat Peter Lieven geschrieben:
>>>>> Hi,
>>>>>
>>>>> I am currently working on optimizing speed for compressed QCOW2
>>>>> images. We use them for templates and would also like to use them for
>>>>> backups, but the latter is almost infeasible because using gzip for
>>>>> compression is horribly slow. I tried to experiment with different
>>>>> options to deflate, but in the end I think its better to use a
>>>>> different compression algorithm for cases where speed matters. As we
>>>>> already have probing for it in configure and as it is widely used I
>>>>> would like to use LZO for that purpose. I think it would be best to
>>>>> have a flag to indicate that compressed blocks use LZO compression,
>>>>> but I would need a little explaination which of the feature fields I
>>>>> have to use to prevent an older (incompatible) Qemu opening LZO
>>>>> compressed QCOW2 images.
>>>>>
>>>>> I also have already some numbers. I converted a fresh Debian 9 Install
>>>>> which has an uncomressed QCOW2 size of 1158 MB with qemu-img to a
>>>>> compressed QCOW2.  With GZIP compression the result is 356MB whereas
>>>>> the LZO version is 452MB. However, the current GZIP variant uses 35
>>>>> seconds for this operation where LZO only needs 4 seconds. I think is
>>>>> is a good trade in especially when its optional so the user can
>>>>> choose.
>>>>>
>>>>> What are your thoughts?
>>>> We had a related RFC patch by Den earlier this year, which never
>>>> received many comment and never got out of RFC:
>>>>
>>>> https://lists.gnu.org/archive/html/qemu-devel/2017-03/msg04682.html
>>> I was not aware of that one. Thanks for pointing out.
>>>
>>>> So he chose a different algorithm (zstd). When I asked, he posted a
>>>> comparison of algorithms (however a generic one and not measured in the
>>>> context of qemu) that suggests that LZO would be slightly faster, but
>>>> have a considerable worse compression ratio with the settings that were
>>>> benchmarked.
>>> My idea to choose LZO was that it is widely available and available in
>>> any distro you can think of. We already have probing for it in configure.
>>> My concern with ZSTD would be that it seems there are no packages
>>> available for most distros and that it seems to be multi-threaded. I
>>> don't
>>> know if this will cause any trouble?
>>>
>> We have had that compression working in multithreaded process.
>>
>>>> I think it's clear that if there is any serious interest in compression,
>>>> we'll want to support at least one more algorithm. What we still need to
>>>> evaluate is which one(s) to take, and whether a simple incompatible flag
>>>> in the header like in Den's patch is enough or whether we should add a
>>>> whole new header field for the compression algorithm (like we already
>>>> have for encryption).
>>> From my side there clearly is interest in optimizing the compression. Its
>>> even possible to speed up zlib by 3-4x times by choosing other parameters
>>> for deflate which unfortunately are not compatible with our inflate
>>> settings.
>>>
>>> I don't know if its worth creating a new header field. Even if we
>>> spent to bits
>>> in the end (one for LZO and one for ZSDT). I think this wouldn't hurt.
>>> However,
>>> there are likely to pop up new compression algorithms in the future and
>>> a header would be more flexible.
>>>
>>> I just don't want to make it too complicated and as you pointed out
>>> compression is
>>> not that interesting for most people - maybe due to its speed.
>>>
>> I think we need something generic but simple. I think that we should not
>> support compression with the different algorithm in the single file.
>>
>> Speaking about compression, we do have different constraints for
>> different situation, f.e. backups are written once and rarely read while
>> generic compression in backing store is read frequently but never
>> read. Thus the exact algorithm should be selectable.
> Pluggable / selectable compression methods are likely the most flexible
> and future-proof. A new header sounds good to me (... said by someone
> who comments on this from the sidelines.)
>
> I would advise caution against multi-threaded compression libraries.
> Unless they are coded very-very carefully with regard to signal handling
> and general error handling / propagation, they cannot be considered
> "opaque" enough.
>
> (I had written and maintained the original (0.x) branch of "lbzip2",
> which was extremely conscious of error handling and signals. That was a
> challenge even in a standalone program, and I didn't even attempt to
> retrofit the code to the existing libbz2 APIs (i.e. I never even tried
> to librarize the code).)
>
> This does not mean that people cannot get such a library right. It's
> just that *by default* such a library will have a number of obscure bugs
> related to: signals, forking, and general error handling. It could also
> have problems with unbounded memory allocation. An MT compression
> library that gets all of this right is the exception IMO, not the norm.
> (I don't know anything about the ZSTD library; it could be such a high
> quality library.)
>
> Another complication with MT *de*compression is that the CPU demand from
> the IO thread (which is by default responsible for handling IO, when not
> using dataplane -- is that right?) would "leak" to other physical
> processors. I believe this can interfere with use cases where people
> carefully isolate host CPUs between "QEMU" and "non-QEMU" workloads,
> plus pin QEMU's VCPU threads, and IO threads, to different host CPUs
> (see vcpupin / emulatorpin / iothreadpin under
> <http://libvirt.org/formatdomain.html#elementsCPUTuning>.) It's probably
> possible to figure out the right thing for "ZSTD threads" as well, but
> IMO it remains a complication nonetheless.
We already have the prototype delegating compression
processing into thread pool processing other IO commands
like flush/fallocate. All compression/decompression
routines in long term MUST be moved out of IO thread
as they add too much latency.

>
> Personally I would recommend a new header, and LZO, as a starting point.
> As pointed out above, LZO is widely available in distros. It has good
> performance, and it is single-threaded similarly to zlib. I use LZO for
> two QEMU-related purposes ATM:
>
> - I use the kdump-lzo format when dumping guest memory
>   (virsh dump $DOMAIN $CORE_FLE --memory-only --format kdump-lzo)
>
> - I use LZO compression for "virsh managedsave"
>   (by setting "save_image_format" in "/etc/libvirt/qemu.conf" to "lzop")
>
> Thanks
> Laszlo

Re: [Qemu-devel] QCOW2 support for LZO compression

[Peter Lieven]

Am 26.06.2017 um 12:16 schrieb Laszlo Ersek:
> On 06/26/17 11:33, Denis V. Lunev wrote:
>> On 06/26/2017 12:20 PM, Peter Lieven wrote:
>>> Am 26.06.2017 um 10:28 schrieb Kevin Wolf:
>>>> [ Cc: qemu-devel; don't post to qemu-block only! ]
>>>>
>>>> Am 26.06.2017 um 09:57 hat Peter Lieven geschrieben:
>>>>> Hi,
>>>>>
>>>>> I am currently working on optimizing speed for compressed QCOW2
>>>>> images. We use them for templates and would also like to use them for
>>>>> backups, but the latter is almost infeasible because using gzip for
>>>>> compression is horribly slow. I tried to experiment with different
>>>>> options to deflate, but in the end I think its better to use a
>>>>> different compression algorithm for cases where speed matters. As we
>>>>> already have probing for it in configure and as it is widely used I
>>>>> would like to use LZO for that purpose. I think it would be best to
>>>>> have a flag to indicate that compressed blocks use LZO compression,
>>>>> but I would need a little explaination which of the feature fields I
>>>>> have to use to prevent an older (incompatible) Qemu opening LZO
>>>>> compressed QCOW2 images.
>>>>>
>>>>> I also have already some numbers. I converted a fresh Debian 9 Install
>>>>> which has an uncomressed QCOW2 size of 1158 MB with qemu-img to a
>>>>> compressed QCOW2.  With GZIP compression the result is 356MB whereas
>>>>> the LZO version is 452MB. However, the current GZIP variant uses 35
>>>>> seconds for this operation where LZO only needs 4 seconds. I think is
>>>>> is a good trade in especially when its optional so the user can
>>>>> choose.
>>>>>
>>>>> What are your thoughts?
>>>> We had a related RFC patch by Den earlier this year, which never
>>>> received many comment and never got out of RFC:
>>>>
>>>> https://lists.gnu.org/archive/html/qemu-devel/2017-03/msg04682.html
>>> I was not aware of that one. Thanks for pointing out.
>>>
>>>> So he chose a different algorithm (zstd). When I asked, he posted a
>>>> comparison of algorithms (however a generic one and not measured in the
>>>> context of qemu) that suggests that LZO would be slightly faster, but
>>>> have a considerable worse compression ratio with the settings that were
>>>> benchmarked.
>>> My idea to choose LZO was that it is widely available and available in
>>> any distro you can think of. We already have probing for it in configure.
>>> My concern with ZSTD would be that it seems there are no packages
>>> available for most distros and that it seems to be multi-threaded. I
>>> don't
>>> know if this will cause any trouble?
>>>
>> We have had that compression working in multithreaded process.
>>
>>>> I think it's clear that if there is any serious interest in compression,
>>>> we'll want to support at least one more algorithm. What we still need to
>>>> evaluate is which one(s) to take, and whether a simple incompatible flag
>>>> in the header like in Den's patch is enough or whether we should add a
>>>> whole new header field for the compression algorithm (like we already
>>>> have for encryption).
>>>  From my side there clearly is interest in optimizing the compression. Its
>>> even possible to speed up zlib by 3-4x times by choosing other parameters
>>> for deflate which unfortunately are not compatible with our inflate
>>> settings.
>>>
>>> I don't know if its worth creating a new header field. Even if we
>>> spent to bits
>>> in the end (one for LZO and one for ZSDT). I think this wouldn't hurt.
>>> However,
>>> there are likely to pop up new compression algorithms in the future and
>>> a header would be more flexible.
>>>
>>> I just don't want to make it too complicated and as you pointed out
>>> compression is
>>> not that interesting for most people - maybe due to its speed.
>>>
>> I think we need something generic but simple. I think that we should not
>> support compression with the different algorithm in the single file.
>>
>> Speaking about compression, we do have different constraints for
>> different situation, f.e. backups are written once and rarely read while
>> generic compression in backing store is read frequently but never
>> read. Thus the exact algorithm should be selectable.
> Pluggable / selectable compression methods are likely the most flexible
> and future-proof. A new header sounds good to me (... said by someone
> who comments on this from the sidelines.)
>
> I would advise caution against multi-threaded compression libraries.
> Unless they are coded very-very carefully with regard to signal handling
> and general error handling / propagation, they cannot be considered
> "opaque" enough.
>
> (I had written and maintained the original (0.x) branch of "lbzip2",
> which was extremely conscious of error handling and signals. That was a
> challenge even in a standalone program, and I didn't even attempt to
> retrofit the code to the existing libbz2 APIs (i.e. I never even tried
> to librarize the code).)
>
> This does not mean that people cannot get such a library right. It's
> just that *by default* such a library will have a number of obscure bugs
> related to: signals, forking, and general error handling. It could also
> have problems with unbounded memory allocation. An MT compression
> library that gets all of this right is the exception IMO, not the norm.
> (I don't know anything about the ZSTD library; it could be such a high
> quality library.)
>
> Another complication with MT *de*compression is that the CPU demand from
> the IO thread (which is by default responsible for handling IO, when not
> using dataplane -- is that right?) would "leak" to other physical
> processors. I believe this can interfere with use cases where people
> carefully isolate host CPUs between "QEMU" and "non-QEMU" workloads,
> plus pin QEMU's VCPU threads, and IO threads, to different host CPUs
> (see vcpupin / emulatorpin / iothreadpin under
> <http://libvirt.org/formatdomain.html#elementsCPUTuning>.) It's probably
> possible to figure out the right thing for "ZSTD threads" as well, but
> IMO it remains a complication nonetheless.
>
> Personally I would recommend a new header, and LZO, as a starting point.
> As pointed out above, LZO is widely available in distros. It has good
> performance, and it is single-threaded similarly to zlib. I use LZO for
> two QEMU-related purposes ATM:
>
> - I use the kdump-lzo format when dumping guest memory
>    (virsh dump $DOMAIN $CORE_FLE --memory-only --format kdump-lzo)
>
> - I use LZO compression for "virsh managedsave"
>    (by setting "save_image_format" in "/etc/libvirt/qemu.conf" to "lzop")

Okay, then let me try to get this build into Qemu. Please bare with me as
I am not familiar with QCOW2 header extensions etc. I would use a char[16]
for the compress algorithm (like it is with the backing_file_format). This would
give us the ability to fail to open a QCOW2 image with a useful error message
like compression algorithm zstd is not supported.

Peter

Re: [Qemu-devel] QCOW2 support for LZO compression

[Kevin Wolf]

Am 26.06.2017 um 11:20 hat Peter Lieven geschrieben:
> >So he chose a different algorithm (zstd). When I asked, he posted a
> >comparison of algorithms (however a generic one and not measured in the
> >context of qemu) that suggests that LZO would be slightly faster, but
> >have a considerable worse compression ratio with the settings that were
> >benchmarked.
> 
> My idea to choose LZO was that it is widely available and available in
> any distro you can think of. We already have probing for it in configure.
> My concern with ZSTD would be that it seems there are no packages
> available for most distros and that it seems to be multi-threaded. I don't
> know if this will cause any trouble?

The availability and that we already link against LZO is a good point. I
think we want to avoid a situation where compressed qcow2 files can't be
read by binaries of popular distributions - after all, downloadable
images are an important use case for compressed images.

> >I think it's clear that if there is any serious interest in compression,
> >we'll want to support at least one more algorithm. What we still need to
> >evaluate is which one(s) to take, and whether a simple incompatible flag
> >in the header like in Den's patch is enough or whether we should add a
> >whole new header field for the compression algorithm (like we already
> >have for encryption).
> 
> From my side there clearly is interest in optimizing the compression. Its
> even possible to speed up zlib by 3-4x times by choosing other parameters
> for deflate which unfortunately are not compatible with our inflate settings.
> 
> I don't know if its worth creating a new header field. Even if we spent to bits
> in the end (one for LZO and one for ZSDT). I think this wouldn't hurt. However,
> there are likely to pop up new compression algorithms in the future and
> a header would be more flexible.

That we could just use different parameters is an important hint that
maybe we don't just need bits to select an algorithm, but also some
space for parameters for the selected algorithm. In this case I think we
need a header change anyway.

> I just don't want to make it too complicated and as you pointed out
> compression is not that interesting for most people - maybe due to its
> speed.

In fact, I think it could be the poor integration in qemu. Compressed
images might be more appealing if you couldn't only write compressed
data with special-case operations like 'qemu-img convert', but just give
an option that you want all writes to be compressed and then use it like
any other image.

Kevin

Re: [Qemu-devel] QCOW2 support for LZO compression

[Peter Lieven]

Am 26.06.2017 um 11:57 schrieb Kevin Wolf:
> Am 26.06.2017 um 11:20 hat Peter Lieven geschrieben:
>>> So he chose a different algorithm (zstd). When I asked, he posted a
>>> comparison of algorithms (however a generic one and not measured in the
>>> context of qemu) that suggests that LZO would be slightly faster, but
>>> have a considerable worse compression ratio with the settings that were
>>> benchmarked.
>> My idea to choose LZO was that it is widely available and available in
>> any distro you can think of. We already have probing for it in configure.
>> My concern with ZSTD would be that it seems there are no packages
>> available for most distros and that it seems to be multi-threaded. I don't
>> know if this will cause any trouble?
> The availability and that we already link against LZO is a good point. I
> think we want to avoid a situation where compressed qcow2 files can't be
> read by binaries of popular distributions - after all, downloadable
> images are an important use case for compressed images.

As long as the default remains gzip I don't see any issues. If you choose
a different algorithm, you should know what you are doing.

>
>>> I think it's clear that if there is any serious interest in compression,
>>> we'll want to support at least one more algorithm. What we still need to
>>> evaluate is which one(s) to take, and whether a simple incompatible flag
>>> in the header like in Den's patch is enough or whether we should add a
>>> whole new header field for the compression algorithm (like we already
>>> have for encryption).
>>  From my side there clearly is interest in optimizing the compression. Its
>> even possible to speed up zlib by 3-4x times by choosing other parameters
>> for deflate which unfortunately are not compatible with our inflate settings.
>>
>> I don't know if its worth creating a new header field. Even if we spent to bits
>> in the end (one for LZO and one for ZSDT). I think this wouldn't hurt. However,
>> there are likely to pop up new compression algorithms in the future and
>> a header would be more flexible.
> That we could just use different parameters is an important hint that
> maybe we don't just need bits to select an algorithm, but also some
> space for parameters for the selected algorithm. In this case I think we
> need a header change anyway.

The issue is, you don't know in advance how much space will be needed for the paramters.
Maybe it would be simpler to have different settings for a compression algorithm threated
as different algorithms. Like zstd-fast, zstd, zstd-slow, gzip, lzo, lzo-slow etc.

>
>> I just don't want to make it too complicated and as you pointed out
>> compression is not that interesting for most people - maybe due to its
>> speed.
> In fact, I think it could be the poor integration in qemu. Compressed
> images might be more appealing if you couldn't only write compressed
> data with special-case operations like 'qemu-img convert', but just give
> an option that you want all writes to be compressed and then use it like
> any other image.

Then we should start to improve that.

Peter

Re: [Qemu-devel] QCOW2 support for LZO compression

[Daniel P. Berrange]

On Mon, Jun 26, 2017 at 12:08:01PM +0200, Peter Lieven wrote:
> Am 26.06.2017 um 11:57 schrieb Kevin Wolf:
> > Am 26.06.2017 um 11:20 hat Peter Lieven geschrieben:
> > > > So he chose a different algorithm (zstd). When I asked, he posted a
> > > > comparison of algorithms (however a generic one and not measured in the
> > > > context of qemu) that suggests that LZO would be slightly faster, but
> > > > have a considerable worse compression ratio with the settings that were
> > > > benchmarked.
> > > My idea to choose LZO was that it is widely available and available in
> > > any distro you can think of. We already have probing for it in configure.
> > > My concern with ZSTD would be that it seems there are no packages
> > > available for most distros and that it seems to be multi-threaded. I don't
> > > know if this will cause any trouble?
> > The availability and that we already link against LZO is a good point. I
> > think we want to avoid a situation where compressed qcow2 files can't be
> > read by binaries of popular distributions - after all, downloadable
> > images are an important use case for compressed images.
> 
> As long as the default remains gzip I don't see any issues. If you choose
> a different algorithm, you should know what you are doing.

The problem comes if Debian were to choose to only link in ZSTD, and RHEL
were choose to only link LZO. Images distributed by one distro, with this
new compression would be unusuable on other distros. So whatever compression
format we choose to add should be something we are confident that all distros
will be happy enabling by default. This favours libraries are already widely
included in distros, especially if QEMU already links to them indirectly.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [Qemu-devel] QCOW2 support for LZO compression

[Peter Lieven]

Am 26.06.2017 um 12:12 schrieb Daniel P. Berrange:
> On Mon, Jun 26, 2017 at 12:08:01PM +0200, Peter Lieven wrote:
>> Am 26.06.2017 um 11:57 schrieb Kevin Wolf:
>>> Am 26.06.2017 um 11:20 hat Peter Lieven geschrieben:
>>>>> So he chose a different algorithm (zstd). When I asked, he posted a
>>>>> comparison of algorithms (however a generic one and not measured in the
>>>>> context of qemu) that suggests that LZO would be slightly faster, but
>>>>> have a considerable worse compression ratio with the settings that were
>>>>> benchmarked.
>>>> My idea to choose LZO was that it is widely available and available in
>>>> any distro you can think of. We already have probing for it in configure.
>>>> My concern with ZSTD would be that it seems there are no packages
>>>> available for most distros and that it seems to be multi-threaded. I don't
>>>> know if this will cause any trouble?
>>> The availability and that we already link against LZO is a good point. I
>>> think we want to avoid a situation where compressed qcow2 files can't be
>>> read by binaries of popular distributions - after all, downloadable
>>> images are an important use case for compressed images.
>> As long as the default remains gzip I don't see any issues. If you choose
>> a different algorithm, you should know what you are doing.
> The problem comes if Debian were to choose to only link in ZSTD, and RHEL
> were choose to only link LZO. Images distributed by one distro, with this
> new compression would be unusuable on other distros. So whatever compression
> format we choose to add should be something we are confident that all distros
> will be happy enabling by default. This favours libraries are already widely
> included in distros, especially if QEMU already links to them indirectly.

This is very strong argument for LZO. I checked with Debian 9 and Ubuntu 16.04
they have liblzo installed in the base system. I think maybe for btrfs compression support.

Peter

Re: [Qemu-devel] QCOW2 support for LZO compression

[Kevin Wolf]

Am 26.06.2017 um 12:08 hat Peter Lieven geschrieben:
> Am 26.06.2017 um 11:57 schrieb Kevin Wolf:
> >Am 26.06.2017 um 11:20 hat Peter Lieven geschrieben:
> >>>So he chose a different algorithm (zstd). When I asked, he posted a
> >>>comparison of algorithms (however a generic one and not measured in the
> >>>context of qemu) that suggests that LZO would be slightly faster, but
> >>>have a considerable worse compression ratio with the settings that were
> >>>benchmarked.
> >>My idea to choose LZO was that it is widely available and available in
> >>any distro you can think of. We already have probing for it in configure.
> >>My concern with ZSTD would be that it seems there are no packages
> >>available for most distros and that it seems to be multi-threaded. I don't
> >>know if this will cause any trouble?
> >The availability and that we already link against LZO is a good point. I
> >think we want to avoid a situation where compressed qcow2 files can't be
> >read by binaries of popular distributions - after all, downloadable
> >images are an important use case for compressed images.
> 
> As long as the default remains gzip I don't see any issues. If you choose
> a different algorithm, you should know what you are doing.

If we add a better compression algorithm, then it's clear to me that the
goal is to make it the default in a year or two.

> >>>I think it's clear that if there is any serious interest in compression,
> >>>we'll want to support at least one more algorithm. What we still need to
> >>>evaluate is which one(s) to take, and whether a simple incompatible flag
> >>>in the header like in Den's patch is enough or whether we should add a
> >>>whole new header field for the compression algorithm (like we already
> >>>have for encryption).
> >> From my side there clearly is interest in optimizing the compression. Its
> >>even possible to speed up zlib by 3-4x times by choosing other parameters
> >>for deflate which unfortunately are not compatible with our inflate settings.
> >>
> >>I don't know if its worth creating a new header field. Even if we spent to bits
> >>in the end (one for LZO and one for ZSDT). I think this wouldn't hurt. However,
> >>there are likely to pop up new compression algorithms in the future and
> >>a header would be more flexible.
> >That we could just use different parameters is an important hint that
> >maybe we don't just need bits to select an algorithm, but also some
> >space for parameters for the selected algorithm. In this case I think we
> >need a header change anyway.
> 
> The issue is, you don't know in advance how much space will be needed
> for the paramters.  Maybe it would be simpler to have different
> settings for a compression algorithm threated as different algorithms.
> Like zstd-fast, zstd, zstd-slow, gzip, lzo, lzo-slow etc.

Yes, how much space is needed for parameters depends on the algorithm.
Normally, when adding an incompatible feature, you would put the new
options inside the main header. However, nothing stops us from using a
header extension instead like we would with compatible features.

So I think this is what we could do:

 * Add a header extension type per algorithm that contains the
   parameters that are needed for this parameter

 * Add a header field that selects the compression algorithm (e.g. a
   32 bit field containing the matching header extension ID). Images
   that contain compression header extensions that don't match this
   header field (or images containing the same compression header
   extension type more than once) are rejected

 * Add an incompatible feature flag that indicates presence of the
   header field + header extension

> >>I just don't want to make it too complicated and as you pointed out
> >>compression is not that interesting for most people - maybe due to its
> >>speed.
> >In fact, I think it could be the poor integration in qemu. Compressed
> >images might be more appealing if you couldn't only write compressed
> >data with special-case operations like 'qemu-img convert', but just give
> >an option that you want all writes to be compressed and then use it like
> >any other image.
> 
> Then we should start to improve that.

Thanks for volunteering. ;-)

Kevin

Re: [Qemu-devel] QCOW2 support for LZO compression

[Peter Lieven]

Am 26.06.2017 um 13:21 schrieb Kevin Wolf:
> Am 26.06.2017 um 12:08 hat Peter Lieven geschrieben:
>> Am 26.06.2017 um 11:57 schrieb Kevin Wolf:
>>> Am 26.06.2017 um 11:20 hat Peter Lieven geschrieben:
>>>>> So he chose a different algorithm (zstd). When I asked, he posted a
>>>>> comparison of algorithms (however a generic one and not measured in the
>>>>> context of qemu) that suggests that LZO would be slightly faster, but
>>>>> have a considerable worse compression ratio with the settings that were
>>>>> benchmarked.
>>>> My idea to choose LZO was that it is widely available and available in
>>>> any distro you can think of. We already have probing for it in configure.
>>>> My concern with ZSTD would be that it seems there are no packages
>>>> available for most distros and that it seems to be multi-threaded. I don't
>>>> know if this will cause any trouble?
>>> The availability and that we already link against LZO is a good point. I
>>> think we want to avoid a situation where compressed qcow2 files can't be
>>> read by binaries of popular distributions - after all, downloadable
>>> images are an important use case for compressed images.
>> As long as the default remains gzip I don't see any issues. If you choose
>> a different algorithm, you should know what you are doing.
> If we add a better compression algorithm, then it's clear to me that the
> goal is to make it the default in a year or two.
>
>>>>> I think it's clear that if there is any serious interest in compression,
>>>>> we'll want to support at least one more algorithm. What we still need to
>>>>> evaluate is which one(s) to take, and whether a simple incompatible flag
>>>>> in the header like in Den's patch is enough or whether we should add a
>>>>> whole new header field for the compression algorithm (like we already
>>>>> have for encryption).
>>>>  From my side there clearly is interest in optimizing the compression. Its
>>>> even possible to speed up zlib by 3-4x times by choosing other parameters
>>>> for deflate which unfortunately are not compatible with our inflate settings.
>>>>
>>>> I don't know if its worth creating a new header field. Even if we spent to bits
>>>> in the end (one for LZO and one for ZSDT). I think this wouldn't hurt. However,
>>>> there are likely to pop up new compression algorithms in the future and
>>>> a header would be more flexible.
>>> That we could just use different parameters is an important hint that
>>> maybe we don't just need bits to select an algorithm, but also some
>>> space for parameters for the selected algorithm. In this case I think we
>>> need a header change anyway.
>> The issue is, you don't know in advance how much space will be needed
>> for the paramters.  Maybe it would be simpler to have different
>> settings for a compression algorithm threated as different algorithms.
>> Like zstd-fast, zstd, zstd-slow, gzip, lzo, lzo-slow etc.
> Yes, how much space is needed for parameters depends on the algorithm.
> Normally, when adding an incompatible feature, you would put the new
> options inside the main header. However, nothing stops us from using a
> header extension instead like we would with compatible features.
>
> So I think this is what we could do:
>
>   * Add a header extension type per algorithm that contains the
>     parameters that are needed for this parameter
>
>   * Add a header field that selects the compression algorithm (e.g. a
>     32 bit field containing the matching header extension ID). Images
>     that contain compression header extensions that don't match this
>     header field (or images containing the same compression header
>     extension type more than once) are rejected
>
>   * Add an incompatible feature flag that indicates presence of the
>     header field + header extension

The header extension per algorithm should be optional. LZO1X for instance
takes no arguments.

>
>>>> I just don't want to make it too complicated and as you pointed out
>>>> compression is not that interesting for most people - maybe due to its
>>>> speed.
>>> In fact, I think it could be the poor integration in qemu. Compressed
>>> images might be more appealing if you couldn't only write compressed
>>> data with special-case operations like 'qemu-img convert', but just give
>>> an option that you want all writes to be compressed and then use it like
>>> any other image.
>> Then we should start to improve that.
> Thanks for volunteering. ;-)

I will try to prepare a patchset for adding LZO support.

Peter

Re: [Qemu-devel] QCOW2 support for LZO compression

[Daniel P. Berrange]

On Mon, Jun 26, 2017 at 11:20:33AM +0200, Peter Lieven wrote:
> 
> Am 26.06.2017 um 10:28 schrieb Kevin Wolf:
> > [ Cc: qemu-devel; don't post to qemu-block only! ]
> > 
> > Am 26.06.2017 um 09:57 hat Peter Lieven geschrieben:
> > > Hi,
> > > 
> > > I am currently working on optimizing speed for compressed QCOW2
> > > images. We use them for templates and would also like to use them for
> > > backups, but the latter is almost infeasible because using gzip for
> > > compression is horribly slow. I tried to experiment with different
> > > options to deflate, but in the end I think its better to use a
> > > different compression algorithm for cases where speed matters. As we
> > > already have probing for it in configure and as it is widely used I
> > > would like to use LZO for that purpose. I think it would be best to
> > > have a flag to indicate that compressed blocks use LZO compression,
> > > but I would need a little explaination which of the feature fields I
> > > have to use to prevent an older (incompatible) Qemu opening LZO
> > > compressed QCOW2 images.
> > > 
> > > I also have already some numbers. I converted a fresh Debian 9 Install
> > > which has an uncomressed QCOW2 size of 1158 MB with qemu-img to a
> > > compressed QCOW2.  With GZIP compression the result is 356MB whereas
> > > the LZO version is 452MB. However, the current GZIP variant uses 35
> > > seconds for this operation where LZO only needs 4 seconds. I think is
> > > is a good trade in especially when its optional so the user can
> > > choose.
> > > 
> > > What are your thoughts?
> > We had a related RFC patch by Den earlier this year, which never
> > received many comment and never got out of RFC:
> > 
> > https://lists.gnu.org/archive/html/qemu-devel/2017-03/msg04682.html
> 
> I was not aware of that one. Thanks for pointing out.
> 
> > 
> > So he chose a different algorithm (zstd). When I asked, he posted a
> > comparison of algorithms (however a generic one and not measured in the
> > context of qemu) that suggests that LZO would be slightly faster, but
> > have a considerable worse compression ratio with the settings that were
> > benchmarked.
> 
> My idea to choose LZO was that it is widely available and available in
> any distro you can think of. We already have probing for it in configure.
> My concern with ZSTD would be that it seems there are no packages
> available for most distros and that it seems to be multi-threaded. I don't
> know if this will cause any trouble?

As a distro maintainer I'd always prefer option to use a library that is
already widely available. While ZSTD could certainly be added to distros,
it means the QEMU maintainer will end up having to package it & become
the defacto long term maintainer of it long term, which is an extra burden.

WRT to making compression algorithms configurable, I think it is important
to ensure we don't add lots of optional algorithms. An important factor is
portability of images - we don't want to end up with each distro's build
of QEMU enabling a different sub-set of compression algorithms, as that is
going to harm interoperability for distributed images. This again makes me
prefer a compression format whose library is widely available, as that makes
it highly likely that the distro will choose to enable it during build.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [Qemu-devel] QCOW2 support for LZO compression

[Denis V. Lunev]

On 06/26/2017 01:04 PM, Daniel P. Berrange wrote:
> On Mon, Jun 26, 2017 at 11:20:33AM +0200, Peter Lieven wrote:
>> Am 26.06.2017 um 10:28 schrieb Kevin Wolf:
>>> [ Cc: qemu-devel; don't post to qemu-block only! ]
>>>
>>> Am 26.06.2017 um 09:57 hat Peter Lieven geschrieben:
>>>> Hi,
>>>>
>>>> I am currently working on optimizing speed for compressed QCOW2
>>>> images. We use them for templates and would also like to use them for
>>>> backups, but the latter is almost infeasible because using gzip for
>>>> compression is horribly slow. I tried to experiment with different
>>>> options to deflate, but in the end I think its better to use a
>>>> different compression algorithm for cases where speed matters. As we
>>>> already have probing for it in configure and as it is widely used I
>>>> would like to use LZO for that purpose. I think it would be best to
>>>> have a flag to indicate that compressed blocks use LZO compression,
>>>> but I would need a little explaination which of the feature fields I
>>>> have to use to prevent an older (incompatible) Qemu opening LZO
>>>> compressed QCOW2 images.
>>>>
>>>> I also have already some numbers. I converted a fresh Debian 9 Install
>>>> which has an uncomressed QCOW2 size of 1158 MB with qemu-img to a
>>>> compressed QCOW2.  With GZIP compression the result is 356MB whereas
>>>> the LZO version is 452MB. However, the current GZIP variant uses 35
>>>> seconds for this operation where LZO only needs 4 seconds. I think is
>>>> is a good trade in especially when its optional so the user can
>>>> choose.
>>>>
>>>> What are your thoughts?
>>> We had a related RFC patch by Den earlier this year, which never
>>> received many comment and never got out of RFC:
>>>
>>> https://lists.gnu.org/archive/html/qemu-devel/2017-03/msg04682.html
>> I was not aware of that one. Thanks for pointing out.
>>
>>> So he chose a different algorithm (zstd). When I asked, he posted a
>>> comparison of algorithms (however a generic one and not measured in the
>>> context of qemu) that suggests that LZO would be slightly faster, but
>>> have a considerable worse compression ratio with the settings that were
>>> benchmarked.
>> My idea to choose LZO was that it is widely available and available in
>> any distro you can think of. We already have probing for it in configure.
>> My concern with ZSTD would be that it seems there are no packages
>> available for most distros and that it seems to be multi-threaded. I don't
>> know if this will cause any trouble?
> As a distro maintainer I'd always prefer option to use a library that is
> already widely available. While ZSTD could certainly be added to distros,
> it means the QEMU maintainer will end up having to package it & become
> the defacto long term maintainer of it long term, which is an extra burden.
>
> WRT to making compression algorithms configurable, I think it is important
> to ensure we don't add lots of optional algorithms. An important factor is
> portability of images - we don't want to end up with each distro's build
> of QEMU enabling a different sub-set of compression algorithms, as that is
> going to harm interoperability for distributed images. This again makes me
> prefer a compression format whose library is widely available, as that makes
> it highly likely that the distro will choose to enable it during build.
>
> Regards,
> Daniel
Speaking about image compression - ZSDT could be questioned
for different things. Though speaking about online migration -
ZSTD IMHO is the best choice. I have started the investigation
keeping that in mind.

Den

Re: [Qemu-devel] QCOW2 support for LZO compression

[Peter Lieven]

Am 26.06.2017 um 12:15 schrieb Denis V. Lunev:
> On 06/26/2017 01:04 PM, Daniel P. Berrange wrote:
>> On Mon, Jun 26, 2017 at 11:20:33AM +0200, Peter Lieven wrote:
>>> Am 26.06.2017 um 10:28 schrieb Kevin Wolf:
>>>> [ Cc: qemu-devel; don't post to qemu-block only! ]
>>>>
>>>> Am 26.06.2017 um 09:57 hat Peter Lieven geschrieben:
>>>>> Hi,
>>>>>
>>>>> I am currently working on optimizing speed for compressed QCOW2
>>>>> images. We use them for templates and would also like to use them for
>>>>> backups, but the latter is almost infeasible because using gzip for
>>>>> compression is horribly slow. I tried to experiment with different
>>>>> options to deflate, but in the end I think its better to use a
>>>>> different compression algorithm for cases where speed matters. As we
>>>>> already have probing for it in configure and as it is widely used I
>>>>> would like to use LZO for that purpose. I think it would be best to
>>>>> have a flag to indicate that compressed blocks use LZO compression,
>>>>> but I would need a little explaination which of the feature fields I
>>>>> have to use to prevent an older (incompatible) Qemu opening LZO
>>>>> compressed QCOW2 images.
>>>>>
>>>>> I also have already some numbers. I converted a fresh Debian 9 Install
>>>>> which has an uncomressed QCOW2 size of 1158 MB with qemu-img to a
>>>>> compressed QCOW2.  With GZIP compression the result is 356MB whereas
>>>>> the LZO version is 452MB. However, the current GZIP variant uses 35
>>>>> seconds for this operation where LZO only needs 4 seconds. I think is
>>>>> is a good trade in especially when its optional so the user can
>>>>> choose.
>>>>>
>>>>> What are your thoughts?
>>>> We had a related RFC patch by Den earlier this year, which never
>>>> received many comment and never got out of RFC:
>>>>
>>>> https://lists.gnu.org/archive/html/qemu-devel/2017-03/msg04682.html
>>> I was not aware of that one. Thanks for pointing out.
>>>
>>>> So he chose a different algorithm (zstd). When I asked, he posted a
>>>> comparison of algorithms (however a generic one and not measured in the
>>>> context of qemu) that suggests that LZO would be slightly faster, but
>>>> have a considerable worse compression ratio with the settings that were
>>>> benchmarked.
>>> My idea to choose LZO was that it is widely available and available in
>>> any distro you can think of. We already have probing for it in configure.
>>> My concern with ZSTD would be that it seems there are no packages
>>> available for most distros and that it seems to be multi-threaded. I don't
>>> know if this will cause any trouble?
>> As a distro maintainer I'd always prefer option to use a library that is
>> already widely available. While ZSTD could certainly be added to distros,
>> it means the QEMU maintainer will end up having to package it & become
>> the defacto long term maintainer of it long term, which is an extra burden.
>>
>> WRT to making compression algorithms configurable, I think it is important
>> to ensure we don't add lots of optional algorithms. An important factor is
>> portability of images - we don't want to end up with each distro's build
>> of QEMU enabling a different sub-set of compression algorithms, as that is
>> going to harm interoperability for distributed images. This again makes me
>> prefer a compression format whose library is widely available, as that makes
>> it highly likely that the distro will choose to enable it during build.
>>
>> Regards,
>> Daniel
> Speaking about image compression - ZSDT could be questioned
> for different things. Though speaking about online migration -
> ZSTD IMHO is the best choice. I have started the investigation
> keeping that in mind.

For migration things a different. In this case you likely control both ends
of the process and use the same distro there.

Peter

Re: [Qemu-devel] QCOW2 support for LZO compression

[Daniel P. Berrange]

On Mon, Jun 26, 2017 at 10:28:38AM +0200, Kevin Wolf wrote:
> [ Cc: qemu-devel; don't post to qemu-block only! ]
> 
> Am 26.06.2017 um 09:57 hat Peter Lieven geschrieben:
> > Hi,
> > 
> > I am currently working on optimizing speed for compressed QCOW2
> > images. We use them for templates and would also like to use them for
> > backups, but the latter is almost infeasible because using gzip for
> > compression is horribly slow. I tried to experiment with different
> > options to deflate, but in the end I think its better to use a
> > different compression algorithm for cases where speed matters. As we
> > already have probing for it in configure and as it is widely used I
> > would like to use LZO for that purpose. I think it would be best to
> > have a flag to indicate that compressed blocks use LZO compression,
> > but I would need a little explaination which of the feature fields I
> > have to use to prevent an older (incompatible) Qemu opening LZO
> > compressed QCOW2 images.
> > 
> > I also have already some numbers. I converted a fresh Debian 9 Install
> > which has an uncomressed QCOW2 size of 1158 MB with qemu-img to a
> > compressed QCOW2.  With GZIP compression the result is 356MB whereas
> > the LZO version is 452MB. However, the current GZIP variant uses 35
> > seconds for this operation where LZO only needs 4 seconds. I think is
> > is a good trade in especially when its optional so the user can
> > choose.
> > 
> > What are your thoughts?
> 
> We had a related RFC patch by Den earlier this year, which never
> received many comment and never got out of RFC:
> 
> https://lists.gnu.org/archive/html/qemu-devel/2017-03/msg04682.html
> 
> So he chose a different algorithm (zstd). When I asked, he posted a
> comparison of algorithms (however a generic one and not measured in the
> context of qemu) that suggests that LZO would be slightly faster, but
> have a considerable worse compression ratio with the settings that were
> benchmarked.
> 
> I think it's clear that if there is any serious interest in compression,
> we'll want to support at least one more algorithm. What we still need to
> evaluate is which one(s) to take, and whether a simple incompatible flag
> in the header like in Den's patch is enough or whether we should add a
> whole new header field for the compression algorithm (like we already
> have for encryption).

We might also want to consider whether doing compression on individual
qcow2 clusters is the best approach for desired usage scenarios, as
compared to adding a general purpose compression block driver in between
the qcow2 driver and the file driver. eg akin to just running 'gzip' over
the entire qcow2 file. The relatively small size of qcow2 clusters limits
the effectiveness of all compression algorithms we might choose between.
By adding a separate compression driver below qcow2, we can choose a much
larger compression block size, independant of qcow2 cluster size.

Taking a random disk image I have with 1 GB of data. If I tell 'xz' to
compress using 65 KB block sizes (to simulate compression attained if
compressing individual qcow2 clusters in isolation), then I get a file
124 MB in size, which is barely better than that attained with qcow2's
built-in gzip compression. If I tell 'xz' to compress with 16 MB block
sizes though, the output is 114 MB in size, which is a massive win over
gzip.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [Qemu-devel] QCOW2 support for LZO compression

[Richard W.M. Jones]

On Mon, Jun 26, 2017 at 12:12:05PM +0100, Daniel P. Berrange wrote:
> On Mon, Jun 26, 2017 at 10:28:38AM +0200, Kevin Wolf wrote:
> > [ Cc: qemu-devel; don't post to qemu-block only! ]
> > 
> > Am 26.06.2017 um 09:57 hat Peter Lieven geschrieben:
> > > Hi,
> > > 
> > > I am currently working on optimizing speed for compressed QCOW2
> > > images. We use them for templates and would also like to use them for
> > > backups, but the latter is almost infeasible because using gzip for
> > > compression is horribly slow. I tried to experiment with different
> > > options to deflate, but in the end I think its better to use a
> > > different compression algorithm for cases where speed matters. As we
> > > already have probing for it in configure and as it is widely used I
> > > would like to use LZO for that purpose. I think it would be best to
> > > have a flag to indicate that compressed blocks use LZO compression,
> > > but I would need a little explaination which of the feature fields I
> > > have to use to prevent an older (incompatible) Qemu opening LZO
> > > compressed QCOW2 images.
> > > 
> > > I also have already some numbers. I converted a fresh Debian 9 Install
> > > which has an uncomressed QCOW2 size of 1158 MB with qemu-img to a
> > > compressed QCOW2.  With GZIP compression the result is 356MB whereas
> > > the LZO version is 452MB. However, the current GZIP variant uses 35
> > > seconds for this operation where LZO only needs 4 seconds. I think is
> > > is a good trade in especially when its optional so the user can
> > > choose.
> > > 
> > > What are your thoughts?
> > 
> > We had a related RFC patch by Den earlier this year, which never
> > received many comment and never got out of RFC:
> > 
> > https://lists.gnu.org/archive/html/qemu-devel/2017-03/msg04682.html
> > 
> > So he chose a different algorithm (zstd). When I asked, he posted a
> > comparison of algorithms (however a generic one and not measured in the
> > context of qemu) that suggests that LZO would be slightly faster, but
> > have a considerable worse compression ratio with the settings that were
> > benchmarked.
> > 
> > I think it's clear that if there is any serious interest in compression,
> > we'll want to support at least one more algorithm. What we still need to
> > evaluate is which one(s) to take, and whether a simple incompatible flag
> > in the header like in Den's patch is enough or whether we should add a
> > whole new header field for the compression algorithm (like we already
> > have for encryption).
> 
> We might also want to consider whether doing compression on individual
> qcow2 clusters is the best approach for desired usage scenarios, as
> compared to adding a general purpose compression block driver in between
> the qcow2 driver and the file driver. eg akin to just running 'gzip' over
> the entire qcow2 file. The relatively small size of qcow2 clusters limits
> the effectiveness of all compression algorithms we might choose between.
> By adding a separate compression driver below qcow2, we can choose a much
> larger compression block size, independant of qcow2 cluster size.
> 
> Taking a random disk image I have with 1 GB of data. If I tell 'xz' to
> compress using 65 KB block sizes (to simulate compression attained if
> compressing individual qcow2 clusters in isolation), then I get a file
> 124 MB in size, which is barely better than that attained with qcow2's
> built-in gzip compression. If I tell 'xz' to compress with 16 MB block
> sizes though, the output is 114 MB in size, which is a massive win over
> gzip.

I think we need both.  qcow2 block compression allows for writable
qcow2 files, whereas xz compression over a whole file is read-only.
Both have real use cases.

liblzma (xz) is a very flexible library which allows for (read-only)
decompression of individual blocks.  If you prepare your xz file with
the --blocksize parameter, keeping it to something small like 16 MB,
then you can make random-access xz-compressed disk images quite
easily.  In fact I wrote code to do exactly that:

https://github.com/libguestfs/nbdkit/tree/master/plugins/xz

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-top is 'top' for virtual machines.  Tiny program with many
powerful monitoring features, net stats, disk stats, logging, etc.
http://people.redhat.com/~rjones/virt-top

Re: [Qemu-devel] QCOW2 support for LZO compression

[Denis V. Lunev]

On 06/26/2017 11:28 AM, Kevin Wolf wrote:
> [ Cc: qemu-devel; don't post to qemu-block only! ]
>
> Am 26.06.2017 um 09:57 hat Peter Lieven geschrieben:
>> Hi,
>>
>> I am currently working on optimizing speed for compressed QCOW2
>> images. We use them for templates and would also like to use them for
>> backups, but the latter is almost infeasible because using gzip for
>> compression is horribly slow. I tried to experiment with different
>> options to deflate, but in the end I think its better to use a
>> different compression algorithm for cases where speed matters. As we
>> already have probing for it in configure and as it is widely used I
>> would like to use LZO for that purpose. I think it would be best to
>> have a flag to indicate that compressed blocks use LZO compression,
>> but I would need a little explaination which of the feature fields I
>> have to use to prevent an older (incompatible) Qemu opening LZO
>> compressed QCOW2 images.
>>
>> I also have already some numbers. I converted a fresh Debian 9 Install
>> which has an uncomressed QCOW2 size of 1158 MB with qemu-img to a
>> compressed QCOW2.  With GZIP compression the result is 356MB whereas
>> the LZO version is 452MB. However, the current GZIP variant uses 35
>> seconds for this operation where LZO only needs 4 seconds. I think is
>> is a good trade in especially when its optional so the user can
>> choose.
>>
>> What are your thoughts?
> We had a related RFC patch by Den earlier this year, which never
> received many comment and never got out of RFC:
>
> https://lists.gnu.org/archive/html/qemu-devel/2017-03/msg04682.html
>
> So he chose a different algorithm (zstd). When I asked, he posted a
> comparison of algorithms (however a generic one and not measured in the
> context of qemu) that suggests that LZO would be slightly faster, but
> have a considerable worse compression ratio with the settings that were
> benchmarked.
>
> I think it's clear that if there is any serious interest in compression,
> we'll want to support at least one more algorithm. What we still need to
> evaluate is which one(s) to take, and whether a simple incompatible flag
> in the header like in Den's patch is enough or whether we should add a
> whole new header field for the compression algorithm (like we already
> have for encryption).
>
> Kevin
I have been contacted today Yann Collet who is ZSTD maintainer, he has
dropped
nowadays status of ZSTD, which could be useful for the discussion:

"_1. zstd package availability_

We have been tracking distribution availability since Zstandard official
release, in September 2016 :
https://github.com/facebook/zstd/issues/320
There is also this tool which tracks availability of packages :
https://repology.org/metapackage/zstd/versions

zstd seems now available as a package in most recent distributions.
It’s even part of “core” for recent BSD releases.
Zstandard v1.0 is still less than 1 year old, so older distributions
typically do not have it (or support a development version).
That’s the main limitation currently. We expect things to improve over time.

 2.

    _Compression speed is good but does not matter
    _For such scenarios, it’s possible to trade speed for more compression.
    At its maximum compression level (--ultra -22), zstd compression
    ratio (and speed) is close to lzma.
    A nice property though is that decompression speed remains roughly
    the same at all compression levels,
    about 10x faster than lzma decompression speed (about 1 GB/s on
    modern CPU).

 3.

    _zstd is multi-threaded, and it’s dangerous_

libzstd is single-threaded.
There is a multi-thread extension, which is enabled in the CLI, but not
in the library.
There is also an experimental target which makes it possible to produce
a MT-enabled library.
Even in this case, the API remains single-threaded by default.
It’s necessary to use dedicated entry points to enable multi-threading.
TL;DR : zstd supports multithreading, but is single threaded by default.

 

 4.

    _How to identify gz format from zstd one ?
    _Many implementations assume they require to add some custom header
    in order to identify gz from zstd.
    That’s not the case: well-formed compression format already provide
    a header with enough information to guarantee their identity.
    Such “good” compression format include gz, zstd, xz, lz4-frame, to
    name a few.
    For zstd, the identifier is a 4-bytes value, documented in the
    compression format :
    https://github.com/facebook/zstd/blob/dev/doc/zstd_compression_format.md#zstandard-frames
    As an example, zstd project provides a zlib-wrapper which is able to
    dynamically recognize an input as gz or zstd, and route to
    appropriate decoder, without any special header :
    https://github.com/facebook/zstd/tree/dev/zlibWrapper


Unfortunately, not all compression algorithm do provide unambiguous
standard header.
LZO, for example, does not by default.
Behind a single name, lzo effectively groups multiple incompatible
variants, which must be correctly identified for proper decoding."

Den

Re: [Qemu-devel] QCOW2 support for LZO compression

[Peter Lieven]

> Am 26.06.2017 um 22:30 schrieb Denis V. Lunev <den@openvz.org>:
> 
>> On 06/26/2017 11:28 AM, Kevin Wolf wrote:
>> [ Cc: qemu-devel; don't post to qemu-block only! ]
>> 
>> Am 26.06.2017 um 09:57 hat Peter Lieven geschrieben:
>>> Hi,
>>> 
>>> I am currently working on optimizing speed for compressed QCOW2
>>> images. We use them for templates and would also like to use them for
>>> backups, but the latter is almost infeasible because using gzip for
>>> compression is horribly slow. I tried to experiment with different
>>> options to deflate, but in the end I think its better to use a
>>> different compression algorithm for cases where speed matters. As we
>>> already have probing for it in configure and as it is widely used I
>>> would like to use LZO for that purpose. I think it would be best to
>>> have a flag to indicate that compressed blocks use LZO compression,
>>> but I would need a little explaination which of the feature fields I
>>> have to use to prevent an older (incompatible) Qemu opening LZO
>>> compressed QCOW2 images.
>>> 
>>> I also have already some numbers. I converted a fresh Debian 9 Install
>>> which has an uncomressed QCOW2 size of 1158 MB with qemu-img to a
>>> compressed QCOW2.  With GZIP compression the result is 356MB whereas
>>> the LZO version is 452MB. However, the current GZIP variant uses 35
>>> seconds for this operation where LZO only needs 4 seconds. I think is
>>> is a good trade in especially when its optional so the user can
>>> choose.
>>> 
>>> What are your thoughts?
>> We had a related RFC patch by Den earlier this year, which never
>> received many comment and never got out of RFC:
>> 
>> https://lists.gnu.org/archive/html/qemu-devel/2017-03/msg04682.html
>> 
>> So he chose a different algorithm (zstd). When I asked, he posted a
>> comparison of algorithms (however a generic one and not measured in the
>> context of qemu) that suggests that LZO would be slightly faster, but
>> have a considerable worse compression ratio with the settings that were
>> benchmarked.
>> 
>> I think it's clear that if there is any serious interest in compression,
>> we'll want to support at least one more algorithm. What we still need to
>> evaluate is which one(s) to take, and whether a simple incompatible flag
>> in the header like in Den's patch is enough or whether we should add a
>> whole new header field for the compression algorithm (like we already
>> have for encryption).
>> 
>> Kevin
> I have been contacted today Yann Collet who is ZSTD maintainer, he has
> dropped
> nowadays status of ZSTD, which could be useful for the discussion:
> 
> "_1. zstd package availability_
> 
> We have been tracking distribution availability since Zstandard official
> release, in September 2016 :
> https://github.com/facebook/zstd/issues/320
> There is also this tool which tracks availability of packages :
> https://repology.org/metapackage/zstd/versions
> 
> zstd seems now available as a package in most recent distributions.
> It’s even part of “core” for recent BSD releases.
> Zstandard v1.0 is still less than 1 year old, so older distributions
> typically do not have it (or support a development version).
> That’s the main limitation currently. We expect things to improve over time.
> 
> 2.
> 
>    _Compression speed is good but does not matter
>    _For such scenarios, it’s possible to trade speed for more compression.
>    At its maximum compression level (--ultra -22), zstd compression
>    ratio (and speed) is close to lzma.
>    A nice property though is that decompression speed remains roughly
>    the same at all compression levels,
>    about 10x faster than lzma decompression speed (about 1 GB/s on
>    modern CPU).
> 
> 3.
> 
>    _zstd is multi-threaded, and it’s dangerous_
> 
> libzstd is single-threaded.
> There is a multi-thread extension, which is enabled in the CLI, but not
> in the library.
> There is also an experimental target which makes it possible to produce
> a MT-enabled library.
> Even in this case, the API remains single-threaded by default.
> It’s necessary to use dedicated entry points to enable multi-threading.
> TL;DR : zstd supports multithreading, but is single threaded by default.
> 
> 
> 
> 4.
> 
>    _How to identify gz format from zstd one ?
>    _Many implementations assume they require to add some custom header
>    in order to identify gz from zstd.
>    That’s not the case: well-formed compression format already provide
>    a header with enough information to guarantee their identity.
>    Such “good” compression format include gz, zstd, xz, lz4-frame, to
>    name a few.
>    For zstd, the identifier is a 4-bytes value, documented in the
>    compression format :
>    https://github.com/facebook/zstd/blob/dev/doc/zstd_compression_format.md#zstandard-frames
>    As an example, zstd project provides a zlib-wrapper which is able to
>    dynamically recognize an input as gz or zstd, and route to
>    appropriate decoder, without any special header :
>    https://github.com/facebook/zstd/tree/dev/zlibWrapper
> 
> 
> Unfortunately, not all compression algorithm do provide unambiguous
> standard header.
> LZO, for example, does not by default.
> Behind a single name, lzo effectively groups multiple incompatible
> variants, which must be correctly identified for proper decoding."
> 
> Den
> 

Hi Den,

thanks for the update.

I am about to have an RFC patchset ready for addition of the compression algorithm header. It will be easy to add support for zstd on top of that.

I would prefer to have the same algorithm for all compressed clusters and avoid detecting for each cluster. If you want to change the algorithm you would have to recode. This way you can also easily detect at open time of the image if you support the compression algorithm and fail directly if necessary.

Peter

Re: [Qemu-devel] QCOW2 support for LZO compression

[Denis V. Lunev]

On 06/26/2017 11:54 PM, Peter Lieven wrote:
>
>> Am 26.06.2017 um 22:30 schrieb Denis V. Lunev <den@openvz.org>:
>>
>>> On 06/26/2017 11:28 AM, Kevin Wolf wrote:
>>> [ Cc: qemu-devel; don't post to qemu-block only! ]
>>>
>>> Am 26.06.2017 um 09:57 hat Peter Lieven geschrieben:
>>>> Hi,
>>>>
>>>> I am currently working on optimizing speed for compressed QCOW2
>>>> images. We use them for templates and would also like to use them for
>>>> backups, but the latter is almost infeasible because using gzip for
>>>> compression is horribly slow. I tried to experiment with different
>>>> options to deflate, but in the end I think its better to use a
>>>> different compression algorithm for cases where speed matters. As we
>>>> already have probing for it in configure and as it is widely used I
>>>> would like to use LZO for that purpose. I think it would be best to
>>>> have a flag to indicate that compressed blocks use LZO compression,
>>>> but I would need a little explaination which of the feature fields I
>>>> have to use to prevent an older (incompatible) Qemu opening LZO
>>>> compressed QCOW2 images.
>>>>
>>>> I also have already some numbers. I converted a fresh Debian 9 Install
>>>> which has an uncomressed QCOW2 size of 1158 MB with qemu-img to a
>>>> compressed QCOW2.  With GZIP compression the result is 356MB whereas
>>>> the LZO version is 452MB. However, the current GZIP variant uses 35
>>>> seconds for this operation where LZO only needs 4 seconds. I think is
>>>> is a good trade in especially when its optional so the user can
>>>> choose.
>>>>
>>>> What are your thoughts?
>>> We had a related RFC patch by Den earlier this year, which never
>>> received many comment and never got out of RFC:
>>>
>>> https://lists.gnu.org/archive/html/qemu-devel/2017-03/msg04682.html
>>>
>>> So he chose a different algorithm (zstd). When I asked, he posted a
>>> comparison of algorithms (however a generic one and not measured in the
>>> context of qemu) that suggests that LZO would be slightly faster, but
>>> have a considerable worse compression ratio with the settings that were
>>> benchmarked.
>>>
>>> I think it's clear that if there is any serious interest in compression,
>>> we'll want to support at least one more algorithm. What we still need to
>>> evaluate is which one(s) to take, and whether a simple incompatible flag
>>> in the header like in Den's patch is enough or whether we should add a
>>> whole new header field for the compression algorithm (like we already
>>> have for encryption).
>>>
>>> Kevin
>> I have been contacted today Yann Collet who is ZSTD maintainer, he has
>> dropped
>> nowadays status of ZSTD, which could be useful for the discussion:
>>
>> "_1. zstd package availability_
>>
>> We have been tracking distribution availability since Zstandard official
>> release, in September 2016 :
>> https://github.com/facebook/zstd/issues/320
>> There is also this tool which tracks availability of packages :
>> https://repology.org/metapackage/zstd/versions
>>
>> zstd seems now available as a package in most recent distributions.
>> It’s even part of “core” for recent BSD releases.
>> Zstandard v1.0 is still less than 1 year old, so older distributions
>> typically do not have it (or support a development version).
>> That’s the main limitation currently. We expect things to improve over time.
>>
>> 2.
>>
>>    _Compression speed is good but does not matter
>>    _For such scenarios, it’s possible to trade speed for more compression.
>>    At its maximum compression level (--ultra -22), zstd compression
>>    ratio (and speed) is close to lzma.
>>    A nice property though is that decompression speed remains roughly
>>    the same at all compression levels,
>>    about 10x faster than lzma decompression speed (about 1 GB/s on
>>    modern CPU).
>>
>> 3.
>>
>>    _zstd is multi-threaded, and it’s dangerous_
>>
>> libzstd is single-threaded.
>> There is a multi-thread extension, which is enabled in the CLI, but not
>> in the library.
>> There is also an experimental target which makes it possible to produce
>> a MT-enabled library.
>> Even in this case, the API remains single-threaded by default.
>> It’s necessary to use dedicated entry points to enable multi-threading.
>> TL;DR : zstd supports multithreading, but is single threaded by default.
>>
>>
>>
>> 4.
>>
>>    _How to identify gz format from zstd one ?
>>    _Many implementations assume they require to add some custom header
>>    in order to identify gz from zstd.
>>    That’s not the case: well-formed compression format already provide
>>    a header with enough information to guarantee their identity.
>>    Such “good” compression format include gz, zstd, xz, lz4-frame, to
>>    name a few.
>>    For zstd, the identifier is a 4-bytes value, documented in the
>>    compression format :
>>    https://github.com/facebook/zstd/blob/dev/doc/zstd_compression_format.md#zstandard-frames
>>    As an example, zstd project provides a zlib-wrapper which is able to
>>    dynamically recognize an input as gz or zstd, and route to
>>    appropriate decoder, without any special header :
>>    https://github.com/facebook/zstd/tree/dev/zlibWrapper
>>
>>
>> Unfortunately, not all compression algorithm do provide unambiguous
>> standard header.
>> LZO, for example, does not by default.
>> Behind a single name, lzo effectively groups multiple incompatible
>> variants, which must be correctly identified for proper decoding."
>>
>> Den
>>
> Hi Den,
>
> thanks for the update.
>
> I am about to have an RFC patchset ready for addition of the compression algorithm header. It will be easy to add support for zstd on top of that.
>
> I would prefer to have the same algorithm for all compressed clusters and avoid detecting for each cluster. If you want to change the algorithm you would have to recode. This way you can also easily detect at open time of the image if you support the compression algorithm and fail directly if necessary.
>
> Peter
completely agree.

Den

Re: [Qemu-devel] QCOW2 support for LZO compression

[Laszlo Ersek]

On 06/26/17 22:30, Denis V. Lunev wrote:

>  3.
> 
>     _zstd is multi-threaded, and it’s dangerous_
> 
> libzstd is single-threaded.
> There is a multi-thread extension, which is enabled in the CLI, but not
> in the library.
> There is also an experimental target which makes it possible to produce
> a MT-enabled library.
> Even in this case, the API remains single-threaded by default.
> It’s necessary to use dedicated entry points to enable multi-threading.
> TL;DR : zstd supports multithreading, but is single threaded by default.

That's a relief (to me anyway).

Thanks
Laszlo