[Qemu-devel] [PATCH] link to .xz files to save some bandwidth

[Qemu-devel] [PATCH] link to .xz files to save some bandwidth

[Paolo Bonzini]

I have converted all .gz and .bz2 files to .xz on download.qemu.org
and this patch would change the links in the website.  This would save
about 5 GB of bandwidth every day (about 20% savings).

xz should be available for all platforms.  Besides providing better
compression ratios, decompression of .xz files is about twice as fast
compared to bzip2.  Compression instead is about 5.5 times slower.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
 _download/source.html   | 4 ++--
 _includes/releases.html | 4 ++--
 2 files changed, 4 insertions(+), 4 deletions(-)

diff --git a/_download/source.html b/_download/source.html
index 1ac8f4f..d090a5e 100644
--- a/_download/source.html
+++ b/_download/source.html
@@ -15,8 +15,8 @@
 
 	{% for release in site.data.releases offset: 0 limit: 1 %}
 	<p>To download and build QEMU {{release.branch}}.{{release.patch}}:</p>
-<pre>wget http://download.qemu-project.org/qemu-{{release.branch}}.{{release.patch}}.tar.bz2
-tar xvjf qemu-{{release.branch}}.{{release.patch}}.tar.bz2
+<pre>wget http://download.qemu-project.org/qemu-{{release.branch}}.{{release.patch}}.tar.xz
+tar xvJf qemu-{{release.branch}}.{{release.patch}}.tar.xz
 cd qemu-{{release.branch}}.{{release.patch}}
 ./configure
 make
diff --git a/_includes/releases.html b/_includes/releases.html
index 2caab8d..226c719 100644
--- a/_includes/releases.html
+++ b/_includes/releases.html
@@ -1,9 +1,9 @@
 <ul>
 {% for release in site.data.releases offset: 0 limit: 4 %}
 <li><strong><a
-  href="http://download.qemu-project.org/qemu-{{release.branch}}.{{release.patch}}.tar.bz2">{{release.branch}}.{{release.patch}}</a></strong>
+  href="http://download.qemu-project.org/qemu-{{release.branch}}.{{release.patch}}.tar.xz">{{release.branch}}.{{release.patch}}</a></strong>
   {{release.date}}<br><a
-  href="http://download.qemu-project.org/qemu-{{release.branch}}.{{release.patch}}.tar.bz2.sig">signature</a> &mdash; <a
+  href="http://download.qemu-project.org/qemu-{{release.branch}}.{{release.patch}}.tar.xz.sig">signature</a> &mdash; <a
   href="http://wiki.qemu-project.org/ChangeLog/{{release.branch}}">changes</a></li>
 {% endfor %}
 </ul>
-- 
2.9.3

Re: [Qemu-devel] [PATCH] link to .xz files to save some bandwidth

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 837 bytes --]

On 02/07/2017 09:59 AM, Paolo Bonzini wrote:
> I have converted all .gz and .bz2 files to .xz on download.qemu.org
> and this patch would change the links in the website.  This would save
> about 5 GB of bandwidth every day (about 20% savings).
> 
> xz should be available for all platforms.  Besides providing better
> compression ratios, decompression of .xz files is about twice as fast
> compared to bzip2.  Compression instead is about 5.5 times slower.

I agree that bzip2 is pointless these days; .gz is a bit more important
when targetting older platforms but I think that does not describe
qemu's target audience.  Do we want to even enter the xz vs. lzma
controversy, or are we okay with being xz-only?

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: [Qemu-devel] [PATCH] link to .xz files to save some bandwidth

[Markus Armbruster]

Eric Blake <eblake@redhat.com> writes:

> On 02/07/2017 09:59 AM, Paolo Bonzini wrote:
>> I have converted all .gz and .bz2 files to .xz on download.qemu.org
>> and this patch would change the links in the website.  This would save
>> about 5 GB of bandwidth every day (about 20% savings).
>> 
>> xz should be available for all platforms.  Besides providing better
>> compression ratios, decompression of .xz files is about twice as fast
>> compared to bzip2.  Compression instead is about 5.5 times slower.
>
> I agree that bzip2 is pointless these days; .gz is a bit more important
> when targetting older platforms but I think that does not describe
> qemu's target audience.  Do we want to even enter the xz vs. lzma
> controversy, or are we okay with being xz-only?

Pointer to / summary of the controversy, so that folks can decide
whether they care?

Re: [Qemu-devel] [PATCH] link to .xz files to save some bandwidth

[Laszlo Ersek]

On 02/08/17 09:49, Markus Armbruster wrote:
> Eric Blake <eblake@redhat.com> writes:
> 
>> On 02/07/2017 09:59 AM, Paolo Bonzini wrote:
>>> I have converted all .gz and .bz2 files to .xz on download.qemu.org
>>> and this patch would change the links in the website.  This would save
>>> about 5 GB of bandwidth every day (about 20% savings).
>>>
>>> xz should be available for all platforms.  Besides providing better
>>> compression ratios, decompression of .xz files is about twice as fast
>>> compared to bzip2.  Compression instead is about 5.5 times slower.
>>
>> I agree that bzip2 is pointless these days; .gz is a bit more important
>> when targetting older platforms but I think that does not describe
>> qemu's target audience.  Do we want to even enter the xz vs. lzma
>> controversy, or are we okay with being xz-only?
> 
> Pointer to / summary of the controversy, so that folks can decide
> whether they care?
> 

http://www.nongnu.org/lzip/xz_inadequate.html

https://www.reddit.com/r/linux/comments/58swzc/xz_format_inadequate_for_longterm_archiving/

Re: [Qemu-devel] [PATCH] link to .xz files to save some bandwidth

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 1321 bytes --]

On 02/08/2017 02:49 AM, Markus Armbruster wrote:
> Eric Blake <eblake@redhat.com> writes:
> 
>> On 02/07/2017 09:59 AM, Paolo Bonzini wrote:
>>> I have converted all .gz and .bz2 files to .xz on download.qemu.org
>>> and this patch would change the links in the website.  This would save
>>> about 5 GB of bandwidth every day (about 20% savings).
>>>
>>> xz should be available for all platforms.  Besides providing better
>>> compression ratios, decompression of .xz files is about twice as fast
>>> compared to bzip2.  Compression instead is about 5.5 times slower.
>>
>> I agree that bzip2 is pointless these days; .gz is a bit more important
>> when targetting older platforms but I think that does not describe
>> qemu's target audience.  Do we want to even enter the xz vs. lzma
>> controversy, or are we okay with being xz-only?
> 
> Pointer to / summary of the controversy, so that folks can decide
> whether they care?

Here's a big thread on the coreutils list, where I proposed a patch to
add lzip distribution alongside xz, but where the patch was more-or-less
dropped due to the debate that ensued:

https://lists.gnu.org/archive/html/coreutils/2017-01/threads.html#00008

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: [Qemu-devel] [PATCH] link to .xz files to save some bandwidth

[Michael Roth]

Quoting Paolo Bonzini (2017-02-07 09:59:56)
> I have converted all .gz and .bz2 files to .xz on download.qemu.org
> and this patch would change the links in the website.  This would save
> about 5 GB of bandwidth every day (about 20% savings).
> 
> xz should be available for all platforms.  Besides providing better
> compression ratios, decompression of .xz files is about twice as fast

> compared to bzip2.  Compression instead is about 5.5 times slower.

I suppose that's a fair enough trade. :)

> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>

Reviewed-by: Michael Roth <mdroth@linux.vnet.ibm.com>

> ---
>  _download/source.html   | 4 ++--
>  _includes/releases.html | 4 ++--

Somewhat related: I think _data/releases.yml needs to be updated as it's
still referencing 2.7.0 instead of 2.7.1. Would one just start updating
that file with each new release, or is there some other process?

>  2 files changed, 4 insertions(+), 4 deletions(-)
> 
> diff --git a/_download/source.html b/_download/source.html
> index 1ac8f4f..d090a5e 100644
> --- a/_download/source.html
> +++ b/_download/source.html
> @@ -15,8 +15,8 @@
> 
>         {% for release in site.data.releases offset: 0 limit: 1 %}
>         <p>To download and build QEMU {{release.branch}}.{{release.patch}}:</p>
> -<pre>wget http://download.qemu-project.org/qemu-{{release.branch}}.{{release.patch}}.tar.bz2
> -tar xvjf qemu-{{release.branch}}.{{release.patch}}.tar.bz2
> +<pre>wget http://download.qemu-project.org/qemu-{{release.branch}}.{{release.patch}}.tar.xz
> +tar xvJf qemu-{{release.branch}}.{{release.patch}}.tar.xz
>  cd qemu-{{release.branch}}.{{release.patch}}
>  ./configure
>  make
> diff --git a/_includes/releases.html b/_includes/releases.html
> index 2caab8d..226c719 100644
> --- a/_includes/releases.html
> +++ b/_includes/releases.html
> @@ -1,9 +1,9 @@
>  <ul>
>  {% for release in site.data.releases offset: 0 limit: 4 %}
>  <li><strong><a
> -  href="http://download.qemu-project.org/qemu-{{release.branch}}.{{release.patch}}.tar.bz2">{{release.branch}}.{{release.patch}}</a></strong>
> +  href="http://download.qemu-project.org/qemu-{{release.branch}}.{{release.patch}}.tar.xz">{{release.branch}}.{{release.patch}}</a></strong>
>    {{release.date}}<br><a
> -  href="http://download.qemu-project.org/qemu-{{release.branch}}.{{release.patch}}.tar.bz2.sig">signature</a> &mdash; <a
> +  href="http://download.qemu-project.org/qemu-{{release.branch}}.{{release.patch}}.tar.xz.sig">signature</a> &mdash; <a
>    href="http://wiki.qemu-project.org/ChangeLog/{{release.branch}}">changes</a></li>
>  {% endfor %}
>  </ul>
> -- 
> 2.9.3
> 
> 

Re: [Qemu-devel] [PATCH] link to .xz files to save some bandwidth

[Paolo Bonzini]

On 08/02/2017 01:43, Michael Roth wrote:
>>  _includes/releases.html | 4 ++--
> 
> Somewhat related: I think _data/releases.yml needs to be updated as it's
> still referencing 2.7.0 instead of 2.7.1. Would one just start updating
> that file with each new release, or is there some other process?

Yes, that's enough.  When you update that file and "git push", all
references are fixed (there are two, in the homepage and the download
page).  Likewise for _data/screenshots.yml.

Paolo

Re: [Qemu-devel] [PATCH] link to .xz files to save some bandwidth

[Laszlo Ersek]

On 02/07/17 16:59, Paolo Bonzini wrote:
> I have converted all .gz and .bz2 files to .xz on download.qemu.org
> and this patch would change the links in the website.  This would save
> about 5 GB of bandwidth every day (about 20% savings).
> 
> xz should be available for all platforms.  Besides providing better
> compression ratios, decompression of .xz files is about twice as fast
> compared to bzip2.  Compression instead is about 5.5 times slower.

Not wanting to waste your time, but did you try decompression with
lbzip2? :) Given the above data (i.e., xz decompression is only twice as
fast as single-threaded bzip2 decompression), if you speed up bzip2
decompression four-fold (hence quad-core), then bzip2 would win by a
factor of two.

In addition, lbzip2 uses a separate, heavily optimized bzip2 compression
algorithm, which (as far as I remember!) is faster than libbz2's
implementation even in single-threaded mode.

(IOW, I disagree with Eric's statement "bzip2 is pointless these days"
-- it is niche, yes, but some uses remain in the multi-core world.)

Here's my completely ad-hoc test.

(0) My laptop is a Lenovo W541, quad core, HT enabled (hence 8 logical
processors, but the extra hyper-threads are practically useless for
multi-threaded compression, because of their contention for the
core-level cache). IOW, my laptop counts as a quad core for the purpose
at hand.

(1) I downloaded <http://download.qemu-project.org/qemu-2.8.0.tar.xz>
from the website, and decompressed it. I didn't measure the time, I just
wanted the TAR file. Size: 177,428,480 bytes.

(2) I re-compressed the TAR with "absurd" XZ compression settings (we're
trying to save server-side upload bandwidth, so maximum compression
settings are justified). Results:

$ time -p xz -9 --extreme --keep --threads=0 qemu-2.8.0.tar
real 83.54
user 83.51
sys 0.07

Output size: 22,337,312 bytes; which is approximately 12.59% of the
original.

(3) Compression with lbzip2 (default is maximum block size, so no need
for "-9"):

$ time -p lbzip2 --keep qemu-2.8.0.tar
real 1.96
user 15.19
sys 0.14

Output size: 28,509,351; which is approximately 16.07% of the original.


So in compression, lbzip2 saves about 3.48% less, relative to the
original, than xz.

In exchange, the CPU time burned for lbzip2 compression is (user + sys
for (3)) / (user + sys for (2)):

  (15.19 + 0.14) / (83.51 + 0.07) ~= 18.34%

of XZ's CPU demand.

And wall clock time for lbzip2 compression (again, using my quad-core
laptop) is:

  1.96 / 83.54 = 2.34%

of XZ's wall clock time.


Let's see decompression:

(4) Decompression with XZ (no multi-threaded decompression is available):

$ time -p xz --decompress --stdout qemu-2.8.0.tar.xz >/dev/null
real 1.39
user 1.38
sys 0.00

(5) Decompression with lbzip2:

$ time -p lbzip2 --decompress --stdout qemu-2.8.0.tar.bz2 >/dev/null
real 0.87
user 6.78
sys 0.06

The CPU demand is significantly higher for lbzip2 decompression:

  (6.78 + 0.06) / 1.38 ~= 495.65%

But the wall clock time is better:

  0.87 / 1.39 ~= 62.59%

(6) Baseline for standard bzip2 decompression (same file from (3)):

$ time -p bzip2 --decompress --stdout qemu-2.8.0.tar.bz2 >/dev/null
real 3.23
user 3.21
sys 0.02

CPU demand relative to XZ decompression:

  (3.21 + 0.02) / 1.38 ~= 234.06%

Wall clock time relative to XZ decompression:

  3.23 / 1.39 ~= 232.37%


So here's the morale that I draw for the BZ2 *format* (for my quad-core
laptop), relative to the XZ format *and* utility:

* Using lbzip2 for compression on the "server side" (assuming my laptop
  is the "server"),
  * we save a whole lot on CPU demand and wall clock time during
    compression, but that's done only once, so it doesn't really matter,
  * we lose 3.48% compression efficiency, which directly translates to
    server upload bandwidth, which may or may not matter.
* Using lbzip2 decompression on the client side (assuming my laptop is
  the client),
  * the CPU demand is almost 5-fold of that of xz decompression,
  * but the wall clock time is approx. 62.59% of that of xz
    decompression.
* Using traditional bzip2 decompression on the client side, it's a pure
  loss:
  * more than 2-fold CPU demand, relative to XZ decompression,
  * more than 2-fold wall clock time, relative to XZ decompression.
* I didn't consider differences in download costs for the clients (I
  think those are negligible).

Thus, if most (interactive) clients use lbzip2 and are at least
quad-core, then the BZ2 format is worth it, for the wall clock time
savings. Otherwise, XZ is better.

... I agree that xz is more widely known and available than lbzip2.

Acked-by: Laszlo Ersek <lersek@redhat.com>

Thanks
Laszlo

> 
> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
> ---
>  _download/source.html   | 4 ++--
>  _includes/releases.html | 4 ++--
>  2 files changed, 4 insertions(+), 4 deletions(-)
> 
> diff --git a/_download/source.html b/_download/source.html
> index 1ac8f4f..d090a5e 100644
> --- a/_download/source.html
> +++ b/_download/source.html
> @@ -15,8 +15,8 @@
>  
>  	{% for release in site.data.releases offset: 0 limit: 1 %}
>  	<p>To download and build QEMU {{release.branch}}.{{release.patch}}:</p>
> -<pre>wget http://download.qemu-project.org/qemu-{{release.branch}}.{{release.patch}}.tar.bz2
> -tar xvjf qemu-{{release.branch}}.{{release.patch}}.tar.bz2
> +<pre>wget http://download.qemu-project.org/qemu-{{release.branch}}.{{release.patch}}.tar.xz
> +tar xvJf qemu-{{release.branch}}.{{release.patch}}.tar.xz
>  cd qemu-{{release.branch}}.{{release.patch}}
>  ./configure
>  make
> diff --git a/_includes/releases.html b/_includes/releases.html
> index 2caab8d..226c719 100644
> --- a/_includes/releases.html
> +++ b/_includes/releases.html
> @@ -1,9 +1,9 @@
>  <ul>
>  {% for release in site.data.releases offset: 0 limit: 4 %}
>  <li><strong><a
> -  href="http://download.qemu-project.org/qemu-{{release.branch}}.{{release.patch}}.tar.bz2">{{release.branch}}.{{release.patch}}</a></strong>
> +  href="http://download.qemu-project.org/qemu-{{release.branch}}.{{release.patch}}.tar.xz">{{release.branch}}.{{release.patch}}</a></strong>
>    {{release.date}}<br><a
> -  href="http://download.qemu-project.org/qemu-{{release.branch}}.{{release.patch}}.tar.bz2.sig">signature</a> &mdash; <a
> +  href="http://download.qemu-project.org/qemu-{{release.branch}}.{{release.patch}}.tar.xz.sig">signature</a> &mdash; <a
>    href="http://wiki.qemu-project.org/ChangeLog/{{release.branch}}">changes</a></li>
>  {% endfor %}
>  </ul>
> 

Re: [Qemu-devel] [PATCH] link to .xz files to save some bandwidth

[Daniel P. Berrange]

On Tue, Feb 07, 2017 at 04:59:56PM +0100, Paolo Bonzini wrote:
> I have converted all .gz and .bz2 files to .xz on download.qemu.org
> and this patch would change the links in the website.  This would save
> about 5 GB of bandwidth every day (about 20% savings).
> 
> xz should be available for all platforms.  Besides providing better
> compression ratios, decompression of .xz files is about twice as fast
> compared to bzip2.  Compression instead is about 5.5 times slower.

I'm not disagreeing with the proposal, but decompression time is
largely irrelevant, IMHO, as any speed up is lost in the noise
once you consider the time to actually build the QEMU after
unpacking it.

Saving GB's of bandwidth a day for our hosting site is a compelling
benefit, if that has direct impact on project running costs. Shorter
download times are nice, if you have a very slow internet link too,
but for most people they'll again be lost in the noise. As discussed
on IRC, a follow up effort would be to change our releases so we only
include QEMU source and ship pre-built ROMs in a separate tar.xz
which would make downloads massively smaller IIUC.


> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
> ---
>  _download/source.html   | 4 ++--
>  _includes/releases.html | 4 ++--
>  2 files changed, 4 insertions(+), 4 deletions(-)
> 
> diff --git a/_download/source.html b/_download/source.html
> index 1ac8f4f..d090a5e 100644
> --- a/_download/source.html
> +++ b/_download/source.html
> @@ -15,8 +15,8 @@
>  
>  	{% for release in site.data.releases offset: 0 limit: 1 %}
>  	<p>To download and build QEMU {{release.branch}}.{{release.patch}}:</p>
> -<pre>wget http://download.qemu-project.org/qemu-{{release.branch}}.{{release.patch}}.tar.bz2
> -tar xvjf qemu-{{release.branch}}.{{release.patch}}.tar.bz2
> +<pre>wget http://download.qemu-project.org/qemu-{{release.branch}}.{{release.patch}}.tar.xz
> +tar xvJf qemu-{{release.branch}}.{{release.patch}}.tar.xz
>  cd qemu-{{release.branch}}.{{release.patch}}
>  ./configure
>  make
> diff --git a/_includes/releases.html b/_includes/releases.html
> index 2caab8d..226c719 100644
> --- a/_includes/releases.html
> +++ b/_includes/releases.html
> @@ -1,9 +1,9 @@
>  <ul>
>  {% for release in site.data.releases offset: 0 limit: 4 %}
>  <li><strong><a
> -  href="http://download.qemu-project.org/qemu-{{release.branch}}.{{release.patch}}.tar.bz2">{{release.branch}}.{{release.patch}}</a></strong>
> +  href="http://download.qemu-project.org/qemu-{{release.branch}}.{{release.patch}}.tar.xz">{{release.branch}}.{{release.patch}}</a></strong>
>    {{release.date}}<br><a
> -  href="http://download.qemu-project.org/qemu-{{release.branch}}.{{release.patch}}.tar.bz2.sig">signature</a> &mdash; <a
> +  href="http://download.qemu-project.org/qemu-{{release.branch}}.{{release.patch}}.tar.xz.sig">signature</a> &mdash; <a
>    href="http://wiki.qemu-project.org/ChangeLog/{{release.branch}}">changes</a></li>
>  {% endfor %}
>  </ul>
> -- 
> 2.9.3
> 
> 

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://entangle-photo.org       -o-    http://search.cpan.org/~danberr/ :|

Re: [Qemu-devel] [PATCH] link to .xz files to save some bandwidth

[Paolo Bonzini]

On 08/02/2017 11:29, Daniel P. Berrange wrote:
> 
> Saving GB's of bandwidth a day for our hosting site is a compelling
> benefit, if that has direct impact on project running costs.

It doesn't but only because we're well within what the host is willing
to donate to QEMU.

Paolo

> Shorter
> download times are nice, if you have a very slow internet link too,
> but for most people they'll again be lost in the noise.