Re: [PATCH] Set XZ_COMPRESSION_LEVEL to -9

[Adrian Bunk <bunk@stusta.de>]

On Fri, Apr 12, 2019 at 02:44:05PM -0700, Andre McCurdy wrote:
> On Fri, Apr 12, 2019 at 2:09 PM Adrian Bunk <bunk@stusta.de> wrote:
> >
> > It is consistent with other compressors also using their
> > best compression.
> >
> > xz is currently the option that offers best compression,
> > using settings even lower than the default is surprising
> > and makes it harder to choose the best available compression.
> >
> > For anyone who cares about compression time using a different
> > option like gzip or bzip2 is the logical choice.
> 
> The concern with xz's highest compression level is memory usage rather
> than CPU time. Quoting from the manpage:
> 
>   Preset   DictSize   CompCPU    CompMem   DecMem
>     -0     256 KiB     0      3 MiB    1 MiB
>     -1       1 MiB     1      9 MiB    2 MiB
>     -2       2 MiB     2     17 MiB    3 MiB
>     -3       4 MiB     3     32 MiB    5 MiB
>     -4       4 MiB     4     48 MiB    5 MiB
>     -5       8 MiB     5     94 MiB    9 MiB
>     -6       8 MiB     6     94 MiB    9 MiB
>     -7      16 MiB     6    186 MiB   17 MiB
>     -8      32 MiB     6    370 MiB   33 MiB
>     -9      64 MiB     6    674 MiB   65 MiB
> 
> Given that these are per-thread the memory requirements may start to
> get a little unreasonable with xz -9. We do limit usage to 50% of
> physical memory via XZ_DEFAULTS though so maybe it's not a big
> concern?

Due to the limit, and also due to the fact that build machines with
very low RAM/core are likely to have bigger problems earlier:

The compile task of a recipe spawns the same number of gcc processes,
and there can be several compile tasks running.

gcc memory usage when compiling C++ code like webkit can be worse than 
these per-thread numbers.

> Either way, perhaps now would be a good time to remove
> XZ_COMPRESSION_LEVEL from image_types.bbclass and instead just include
> the compression level option in XZ_DEFAULTS so that it gets used
> consistently everywhere.

OPKGBUILDCMD uses XZ_DEFAULTS, and the decompression memory requirement 
on the target for -9 might be a problem if someone uses package feeds
on low-memory targets.

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed