[PATCH] Set XZ_COMPRESSION_LEVEL to -9

[PATCH] Set XZ_COMPRESSION_LEVEL to -9

[Adrian Bunk]

It is consistent with other compressors also using their
best compression.

xz is currently the option that offers best compression,
using settings even lower than the default is surprising
and makes it harder to choose the best available compression.

For anyone who cares about compression time using a different
option like gzip or bzip2 is the logical choice.

For decompression, better compressed is actually faster
to decompress.

Signed-off-by: Adrian Bunk <bunk@stusta.de>
---
 meta/classes/image_types.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/classes/image_types.bbclass b/meta/classes/image_types.bbclass
index 95aa1674cf..1c44ec4a80 100644
--- a/meta/classes/image_types.bbclass
+++ b/meta/classes/image_types.bbclass
@@ -54,7 +54,7 @@ def imagetypes_getdepends(d):
     # Sort the set so that ordering is consistant
     return " ".join(sorted(deps))
 
-XZ_COMPRESSION_LEVEL ?= "-3"
+XZ_COMPRESSION_LEVEL ?= "-9"
 XZ_INTEGRITY_CHECK ?= "crc32"
 
 ZIP_COMPRESSION_LEVEL ?= "-9"
-- 
2.17.1

Re: [PATCH] Set XZ_COMPRESSION_LEVEL to -9

[Andre McCurdy]

On Fri, Apr 12, 2019 at 2:09 PM Adrian Bunk <bunk@stusta.de> wrote:
>
> It is consistent with other compressors also using their
> best compression.
>
> xz is currently the option that offers best compression,
> using settings even lower than the default is surprising
> and makes it harder to choose the best available compression.
>
> For anyone who cares about compression time using a different
> option like gzip or bzip2 is the logical choice.

The concern with xz's highest compression level is memory usage rather
than CPU time. Quoting from the manpage:

  Preset   DictSize   CompCPU    CompMem   DecMem
    -0     256 KiB     0      3 MiB    1 MiB
    -1       1 MiB     1      9 MiB    2 MiB
    -2       2 MiB     2     17 MiB    3 MiB
    -3       4 MiB     3     32 MiB    5 MiB
    -4       4 MiB     4     48 MiB    5 MiB
    -5       8 MiB     5     94 MiB    9 MiB
    -6       8 MiB     6     94 MiB    9 MiB
    -7      16 MiB     6    186 MiB   17 MiB
    -8      32 MiB     6    370 MiB   33 MiB
    -9      64 MiB     6    674 MiB   65 MiB

Given that these are per-thread the memory requirements may start to
get a little unreasonable with xz -9. We do limit usage to 50% of
physical memory via XZ_DEFAULTS though so maybe it's not a big
concern?

Either way, perhaps now would be a good time to remove
XZ_COMPRESSION_LEVEL from image_types.bbclass and instead just include
the compression level option in XZ_DEFAULTS so that it gets used
consistently everywhere.

> For decompression, better compressed is actually faster
> to decompress.
>
> Signed-off-by: Adrian Bunk <bunk@stusta.de>
> ---
>  meta/classes/image_types.bbclass | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/meta/classes/image_types.bbclass b/meta/classes/image_types.bbclass
> index 95aa1674cf..1c44ec4a80 100644
> --- a/meta/classes/image_types.bbclass
> +++ b/meta/classes/image_types.bbclass
> @@ -54,7 +54,7 @@ def imagetypes_getdepends(d):
>      # Sort the set so that ordering is consistant
>      return " ".join(sorted(deps))
>
> -XZ_COMPRESSION_LEVEL ?= "-3"
> +XZ_COMPRESSION_LEVEL ?= "-9"
>  XZ_INTEGRITY_CHECK ?= "crc32"
>
>  ZIP_COMPRESSION_LEVEL ?= "-9"
> --
> 2.17.1
>
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [PATCH] Set XZ_COMPRESSION_LEVEL to -9

[Khem Raj]

On Fri, Apr 12, 2019 at 2:09 PM Adrian Bunk <bunk@stusta.de> wrote:
>
> It is consistent with other compressors also using their
> best compression.
>
> xz is currently the option that offers best compression,
> using settings even lower than the default is surprising
> and makes it harder to choose the best available compression.
>
> For anyone who cares about compression time using a different
> option like gzip or bzip2 is the logical choice.
>
> For decompression, better compressed is actually faster
> to decompress.

I worry about resources we need to spend when creating these archives.
Can you post some comparisons
along to assess the performance.

>
> Signed-off-by: Adrian Bunk <bunk@stusta.de>
> ---
>  meta/classes/image_types.bbclass | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/meta/classes/image_types.bbclass b/meta/classes/image_types.bbclass
> index 95aa1674cf..1c44ec4a80 100644
> --- a/meta/classes/image_types.bbclass
> +++ b/meta/classes/image_types.bbclass
> @@ -54,7 +54,7 @@ def imagetypes_getdepends(d):
>      # Sort the set so that ordering is consistant
>      return " ".join(sorted(deps))
>
> -XZ_COMPRESSION_LEVEL ?= "-3"
> +XZ_COMPRESSION_LEVEL ?= "-9"
>  XZ_INTEGRITY_CHECK ?= "crc32"
>
>  ZIP_COMPRESSION_LEVEL ?= "-9"
> --
> 2.17.1
>
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [PATCH] Set XZ_COMPRESSION_LEVEL to -9

[Adrian Bunk]

On Fri, Apr 12, 2019 at 02:44:05PM -0700, Andre McCurdy wrote:
> On Fri, Apr 12, 2019 at 2:09 PM Adrian Bunk <bunk@stusta.de> wrote:
> >
> > It is consistent with other compressors also using their
> > best compression.
> >
> > xz is currently the option that offers best compression,
> > using settings even lower than the default is surprising
> > and makes it harder to choose the best available compression.
> >
> > For anyone who cares about compression time using a different
> > option like gzip or bzip2 is the logical choice.
> 
> The concern with xz's highest compression level is memory usage rather
> than CPU time. Quoting from the manpage:
> 
>   Preset   DictSize   CompCPU    CompMem   DecMem
>     -0     256 KiB     0      3 MiB    1 MiB
>     -1       1 MiB     1      9 MiB    2 MiB
>     -2       2 MiB     2     17 MiB    3 MiB
>     -3       4 MiB     3     32 MiB    5 MiB
>     -4       4 MiB     4     48 MiB    5 MiB
>     -5       8 MiB     5     94 MiB    9 MiB
>     -6       8 MiB     6     94 MiB    9 MiB
>     -7      16 MiB     6    186 MiB   17 MiB
>     -8      32 MiB     6    370 MiB   33 MiB
>     -9      64 MiB     6    674 MiB   65 MiB
> 
> Given that these are per-thread the memory requirements may start to
> get a little unreasonable with xz -9. We do limit usage to 50% of
> physical memory via XZ_DEFAULTS though so maybe it's not a big
> concern?

Due to the limit, and also due to the fact that build machines with
very low RAM/core are likely to have bigger problems earlier:

The compile task of a recipe spawns the same number of gcc processes,
and there can be several compile tasks running.

gcc memory usage when compiling C++ code like webkit can be worse than 
these per-thread numbers.

> Either way, perhaps now would be a good time to remove
> XZ_COMPRESSION_LEVEL from image_types.bbclass and instead just include
> the compression level option in XZ_DEFAULTS so that it gets used
> consistently everywhere.

OPKGBUILDCMD uses XZ_DEFAULTS, and the decompression memory requirement 
on the target for -9 might be a problem if someone uses package feeds
on low-memory targets.

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: [PATCH] Set XZ_COMPRESSION_LEVEL to -9

[Adrian Bunk]

On Fri, Apr 12, 2019 at 04:12:19PM -0700, Khem Raj wrote:
> On Fri, Apr 12, 2019 at 2:09 PM Adrian Bunk <bunk@stusta.de> wrote:
> >
> > It is consistent with other compressors also using their
> > best compression.
> >
> > xz is currently the option that offers best compression,
> > using settings even lower than the default is surprising
> > and makes it harder to choose the best available compression.
> >
> > For anyone who cares about compression time using a different
> > option like gzip or bzip2 is the logical choice.
> >
> > For decompression, better compressed is actually faster
> > to decompress.
> 
> I worry about resources we need to spend when creating these archives.
> Can you post some comparisons
> along to assess the performance.

On my laptop single-threaded compression of an image I am frequently
building takes 20s with -3 and 55s with -9.

Building the whole image from scratch takes more than 2 hours.

If you care about resources spent, I remember it saved me around 10 
minutes build time when I changed OPKGBUILDCMD from the default "xz -6" 
to "gzip -1" - for the common case where package feeds are not used 
there is no point in spending time on compression and decompression.

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed