* [PATCH] Set XZ_COMPRESSION_LEVEL to -9
@ 2019-04-12 21:09 Adrian Bunk
2019-04-12 21:44 ` Andre McCurdy
2019-04-12 23:12 ` Khem Raj
0 siblings, 2 replies; 5+ messages in thread
From: Adrian Bunk @ 2019-04-12 21:09 UTC (permalink / raw)
To: openembedded-core
It is consistent with other compressors also using their
best compression.
xz is currently the option that offers best compression,
using settings even lower than the default is surprising
and makes it harder to choose the best available compression.
For anyone who cares about compression time using a different
option like gzip or bzip2 is the logical choice.
For decompression, better compressed is actually faster
to decompress.
Signed-off-by: Adrian Bunk <bunk@stusta.de>
---
meta/classes/image_types.bbclass | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/meta/classes/image_types.bbclass b/meta/classes/image_types.bbclass
index 95aa1674cf..1c44ec4a80 100644
--- a/meta/classes/image_types.bbclass
+++ b/meta/classes/image_types.bbclass
@@ -54,7 +54,7 @@ def imagetypes_getdepends(d):
# Sort the set so that ordering is consistant
return " ".join(sorted(deps))
-XZ_COMPRESSION_LEVEL ?= "-3"
+XZ_COMPRESSION_LEVEL ?= "-9"
XZ_INTEGRITY_CHECK ?= "crc32"
ZIP_COMPRESSION_LEVEL ?= "-9"
--
2.17.1
^ permalink raw reply related [flat|nested] 5+ messages in thread* Re: [PATCH] Set XZ_COMPRESSION_LEVEL to -9
2019-04-12 21:09 [PATCH] Set XZ_COMPRESSION_LEVEL to -9 Adrian Bunk
@ 2019-04-12 21:44 ` Andre McCurdy
2019-04-13 6:17 ` Adrian Bunk
2019-04-12 23:12 ` Khem Raj
1 sibling, 1 reply; 5+ messages in thread
From: Andre McCurdy @ 2019-04-12 21:44 UTC (permalink / raw)
To: Adrian Bunk; +Cc: OE Core mailing list
On Fri, Apr 12, 2019 at 2:09 PM Adrian Bunk <bunk@stusta.de> wrote:
>
> It is consistent with other compressors also using their
> best compression.
>
> xz is currently the option that offers best compression,
> using settings even lower than the default is surprising
> and makes it harder to choose the best available compression.
>
> For anyone who cares about compression time using a different
> option like gzip or bzip2 is the logical choice.
The concern with xz's highest compression level is memory usage rather
than CPU time. Quoting from the manpage:
Preset DictSize CompCPU CompMem DecMem
-0 256 KiB 0 3 MiB 1 MiB
-1 1 MiB 1 9 MiB 2 MiB
-2 2 MiB 2 17 MiB 3 MiB
-3 4 MiB 3 32 MiB 5 MiB
-4 4 MiB 4 48 MiB 5 MiB
-5 8 MiB 5 94 MiB 9 MiB
-6 8 MiB 6 94 MiB 9 MiB
-7 16 MiB 6 186 MiB 17 MiB
-8 32 MiB 6 370 MiB 33 MiB
-9 64 MiB 6 674 MiB 65 MiB
Given that these are per-thread the memory requirements may start to
get a little unreasonable with xz -9. We do limit usage to 50% of
physical memory via XZ_DEFAULTS though so maybe it's not a big
concern?
Either way, perhaps now would be a good time to remove
XZ_COMPRESSION_LEVEL from image_types.bbclass and instead just include
the compression level option in XZ_DEFAULTS so that it gets used
consistently everywhere.
> For decompression, better compressed is actually faster
> to decompress.
>
> Signed-off-by: Adrian Bunk <bunk@stusta.de>
> ---
> meta/classes/image_types.bbclass | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/meta/classes/image_types.bbclass b/meta/classes/image_types.bbclass
> index 95aa1674cf..1c44ec4a80 100644
> --- a/meta/classes/image_types.bbclass
> +++ b/meta/classes/image_types.bbclass
> @@ -54,7 +54,7 @@ def imagetypes_getdepends(d):
> # Sort the set so that ordering is consistant
> return " ".join(sorted(deps))
>
> -XZ_COMPRESSION_LEVEL ?= "-3"
> +XZ_COMPRESSION_LEVEL ?= "-9"
> XZ_INTEGRITY_CHECK ?= "crc32"
>
> ZIP_COMPRESSION_LEVEL ?= "-9"
> --
> 2.17.1
>
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
^ permalink raw reply [flat|nested] 5+ messages in thread* Re: [PATCH] Set XZ_COMPRESSION_LEVEL to -9
2019-04-12 21:44 ` Andre McCurdy
@ 2019-04-13 6:17 ` Adrian Bunk
0 siblings, 0 replies; 5+ messages in thread
From: Adrian Bunk @ 2019-04-13 6:17 UTC (permalink / raw)
To: Andre McCurdy; +Cc: OE Core mailing list
On Fri, Apr 12, 2019 at 02:44:05PM -0700, Andre McCurdy wrote:
> On Fri, Apr 12, 2019 at 2:09 PM Adrian Bunk <bunk@stusta.de> wrote:
> >
> > It is consistent with other compressors also using their
> > best compression.
> >
> > xz is currently the option that offers best compression,
> > using settings even lower than the default is surprising
> > and makes it harder to choose the best available compression.
> >
> > For anyone who cares about compression time using a different
> > option like gzip or bzip2 is the logical choice.
>
> The concern with xz's highest compression level is memory usage rather
> than CPU time. Quoting from the manpage:
>
> Preset DictSize CompCPU CompMem DecMem
> -0 256 KiB 0 3 MiB 1 MiB
> -1 1 MiB 1 9 MiB 2 MiB
> -2 2 MiB 2 17 MiB 3 MiB
> -3 4 MiB 3 32 MiB 5 MiB
> -4 4 MiB 4 48 MiB 5 MiB
> -5 8 MiB 5 94 MiB 9 MiB
> -6 8 MiB 6 94 MiB 9 MiB
> -7 16 MiB 6 186 MiB 17 MiB
> -8 32 MiB 6 370 MiB 33 MiB
> -9 64 MiB 6 674 MiB 65 MiB
>
> Given that these are per-thread the memory requirements may start to
> get a little unreasonable with xz -9. We do limit usage to 50% of
> physical memory via XZ_DEFAULTS though so maybe it's not a big
> concern?
Due to the limit, and also due to the fact that build machines with
very low RAM/core are likely to have bigger problems earlier:
The compile task of a recipe spawns the same number of gcc processes,
and there can be several compile tasks running.
gcc memory usage when compiling C++ code like webkit can be worse than
these per-thread numbers.
> Either way, perhaps now would be a good time to remove
> XZ_COMPRESSION_LEVEL from image_types.bbclass and instead just include
> the compression level option in XZ_DEFAULTS so that it gets used
> consistently everywhere.
OPKGBUILDCMD uses XZ_DEFAULTS, and the decompression memory requirement
on the target for -9 might be a problem if someone uses package feeds
on low-memory targets.
cu
Adrian
--
"Is there not promise of rain?" Ling Tan asked suddenly out
of the darkness. There had been need of rain for many days.
"Only a promise," Lao Er said.
Pearl S. Buck - Dragon Seed
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH] Set XZ_COMPRESSION_LEVEL to -9
2019-04-12 21:09 [PATCH] Set XZ_COMPRESSION_LEVEL to -9 Adrian Bunk
2019-04-12 21:44 ` Andre McCurdy
@ 2019-04-12 23:12 ` Khem Raj
2019-04-13 6:52 ` Adrian Bunk
1 sibling, 1 reply; 5+ messages in thread
From: Khem Raj @ 2019-04-12 23:12 UTC (permalink / raw)
To: Adrian Bunk; +Cc: Patches and discussions about the oe-core layer
On Fri, Apr 12, 2019 at 2:09 PM Adrian Bunk <bunk@stusta.de> wrote:
>
> It is consistent with other compressors also using their
> best compression.
>
> xz is currently the option that offers best compression,
> using settings even lower than the default is surprising
> and makes it harder to choose the best available compression.
>
> For anyone who cares about compression time using a different
> option like gzip or bzip2 is the logical choice.
>
> For decompression, better compressed is actually faster
> to decompress.
I worry about resources we need to spend when creating these archives.
Can you post some comparisons
along to assess the performance.
>
> Signed-off-by: Adrian Bunk <bunk@stusta.de>
> ---
> meta/classes/image_types.bbclass | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/meta/classes/image_types.bbclass b/meta/classes/image_types.bbclass
> index 95aa1674cf..1c44ec4a80 100644
> --- a/meta/classes/image_types.bbclass
> +++ b/meta/classes/image_types.bbclass
> @@ -54,7 +54,7 @@ def imagetypes_getdepends(d):
> # Sort the set so that ordering is consistant
> return " ".join(sorted(deps))
>
> -XZ_COMPRESSION_LEVEL ?= "-3"
> +XZ_COMPRESSION_LEVEL ?= "-9"
> XZ_INTEGRITY_CHECK ?= "crc32"
>
> ZIP_COMPRESSION_LEVEL ?= "-9"
> --
> 2.17.1
>
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH] Set XZ_COMPRESSION_LEVEL to -9
2019-04-12 23:12 ` Khem Raj
@ 2019-04-13 6:52 ` Adrian Bunk
0 siblings, 0 replies; 5+ messages in thread
From: Adrian Bunk @ 2019-04-13 6:52 UTC (permalink / raw)
To: Khem Raj; +Cc: Patches and discussions about the oe-core layer
On Fri, Apr 12, 2019 at 04:12:19PM -0700, Khem Raj wrote:
> On Fri, Apr 12, 2019 at 2:09 PM Adrian Bunk <bunk@stusta.de> wrote:
> >
> > It is consistent with other compressors also using their
> > best compression.
> >
> > xz is currently the option that offers best compression,
> > using settings even lower than the default is surprising
> > and makes it harder to choose the best available compression.
> >
> > For anyone who cares about compression time using a different
> > option like gzip or bzip2 is the logical choice.
> >
> > For decompression, better compressed is actually faster
> > to decompress.
>
> I worry about resources we need to spend when creating these archives.
> Can you post some comparisons
> along to assess the performance.
On my laptop single-threaded compression of an image I am frequently
building takes 20s with -3 and 55s with -9.
Building the whole image from scratch takes more than 2 hours.
If you care about resources spent, I remember it saved me around 10
minutes build time when I changed OPKGBUILDCMD from the default "xz -6"
to "gzip -1" - for the common case where package feeds are not used
there is no point in spending time on compression and decompression.
cu
Adrian
--
"Is there not promise of rain?" Ling Tan asked suddenly out
of the darkness. There had been need of rain for many days.
"Only a promise," Lao Er said.
Pearl S. Buck - Dragon Seed
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2019-04-13 6:52 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2019-04-12 21:09 [PATCH] Set XZ_COMPRESSION_LEVEL to -9 Adrian Bunk
2019-04-12 21:44 ` Andre McCurdy
2019-04-13 6:17 ` Adrian Bunk
2019-04-12 23:12 ` Khem Raj
2019-04-13 6:52 ` Adrian Bunk
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox