Re: Re: Re: Re: [OE-core] [PATCH] opkg: enable zstd support

[Alex Stewart <alex.stewart@ni.com>]

Thanks for doing the research.

What was the build time on the zstd L19 compression vs. xz? Was there 
still an improvement?

On 9/28/22 11:50, Etienne Cordonnier wrote:
> I tested it a bit more today. I used the standard poky's local.conf 
> and added those lines. You can see that at zstd level 9 there is still 
> a significant difference in the compression ratio with xz for gcc-dbg 
> which is a big file. At zstd level 19, gcc-dbg is 241MB big instead of 
> 214MB with xz (12.6% more).
>
> PACKAGE_CLASSES = "package_ipk"
> ZSTD_DEFAULTS = "-T0 -19"
> PACKAGECONFIG:append:pn-opkg-native = " zstd"
> OPKGBUILDCMD = "opkg-build -Z zstd -a ${ZSTD_DEFAULTS}"
>
> Zstd at level 9:
>
> build$ ls -lh tmp-zstd-level9/deploy/ipk/core2-64/  --sort=size | head 
> -n10
> total 1.5G
> -rw-r--r-- 3 ecordonnier ecordonnier  302M Sep 28 18:33 
> gcc-dbg_12.2.0-r0_core2-64.ipk
> -rw-r--r-- 3 ecordonnier ecordonnier  249M Sep 28 18:33 
> openssl-dbg_3.0.5-r0_core2-64.ipk
> -rw-r--r-- 3 ecordonnier ecordonnier  106M Sep 28 18:33 
> openssl-ptest_3.0.5-r0_core2-64.ipk
> -rw-r--r-- 3 ecordonnier ecordonnier   44M Sep 28 18:33 
> binutils-dbg_2.39-r0_core2-64.ipk
> -rw-r--r-- 3 ecordonnier ecordonnier   26M Sep 28 18:33 
> perl-ptest_5.36.0-r0_core2-64.ipk
> -rw-r--r-- 3 ecordonnier ecordonnier   23M Sep 28 18:33 
> gcc_12.2.0-r0_core2-64.ipk
> -rw-r--r-- 3 ecordonnier ecordonnier   15M Sep 28 18:32 
> elfutils-ptest_0.187-r0_core2-64.ipk
> -rw-r--r-- 3 ecordonnier ecordonnier   15M Sep 28 18:33 
> gcc-src_12.2.0-r0_core2-64.ipk
> -rw-r--r-- 3 ecordonnier ecordonnier   13M Sep 28 18:33 
> coreutils-dbg_9.1-r0_core2-64.ipk
>
> zstd at level 19:
>
> build$ ls -lh tmp-zstd-level19/deploy/ipk/core2-64/  --sort=size | 
> head -n10
> total 1.1G
> -rw-r--r-- 3 ecordonnier ecordonnier  241M Sep 28 18:04 
> gcc-dbg_12.2.0-r0_core2-64.ipk
> -rw-r--r-- 3 ecordonnier ecordonnier  213M Sep 28 18:03 
> openssl-dbg_3.0.5-r0_core2-64.ipk
> -rw-r--r-- 3 ecordonnier ecordonnier   36M Sep 28 17:58 
> binutils-dbg_2.39-r0_core2-64.ipk
> -rw-r--r-- 3 ecordonnier ecordonnier   30M Sep 28 18:00 
> openssl-ptest_3.0.5-r0_core2-64.ipk
> -rw-r--r-- 3 ecordonnier ecordonnier   22M Sep 28 17:59 
> perl-ptest_5.36.0-r0_core2-64.ipk
> -rw-r--r-- 3 ecordonnier ecordonnier   20M Sep 28 17:58 
> gcc_12.2.0-r0_core2-64.ipk
> -rw-r--r-- 3 ecordonnier ecordonnier   14M Sep 28 17:57 
> elfutils-ptest_0.187-r0_core2-64.ipk
> -rw-r--r-- 3 ecordonnier ecordonnier   12M Sep 28 17:58 
> gcc-src_12.2.0-r0_core2-64.ipk
> -rw-r--r-- 3 ecordonnier ecordonnier   12M Sep 28 17:57 
> g++_12.2.0-r0_core2-64.ipk
>
> xz at level 9 (the default):
>
> build$ ls -lh tmp-xz/deploy/ipk/core2-64/  --sort=size | head -n10
> total 963M
> -rw-r--r-- 3 ecordonnier ecordonnier  214M Sep 14 10:44 
> gcc-dbg_12.2.0-r0_core2-64.ipk
> -rw-r--r-- 3 ecordonnier ecordonnier  193M Sep 14 10:44 
> openssl-dbg_3.0.5-r0_core2-64.ipk
> -rw-r--r-- 3 ecordonnier ecordonnier   35M Sep 14 10:44 
> openssl-ptest_3.0.5-r0_core2-64.ipk
> -rw-r--r-- 3 ecordonnier ecordonnier   32M Sep 14 10:44 
> binutils-dbg_2.39-r0_core2-64.ipk
> -rw-r--r-- 3 ecordonnier ecordonnier   20M Sep 14 10:44 
> perl-ptest_5.36.0-r0_core2-64.ipk
> -rw-r--r-- 3 ecordonnier ecordonnier   19M Sep 14 10:44 
> gcc_12.2.0-r0_core2-64.ipk
> -rw-r--r-- 3 ecordonnier ecordonnier   13M Sep 14 10:44 
> elfutils-ptest_0.187-r0_core2-64.ipk
> -rw-r--r-- 3 ecordonnier ecordonnier   12M Sep 14 10:44 
> gcc-src_12.2.0-r0_core2-64.ipk
> -rw-r--r-- 3 ecordonnier ecordonnier   11M Sep 14 10:44 
> g++_12.2.0-r0_core2-64.ipk
>
> On Wed, Sep 14, 2022 at 5:42 PM Khem Raj <raj.khem@gmail.com> wrote:
>
>     On Wed, Sep 14, 2022 at 8:37 AM Alex Stewart <alex.stewart@ni.com>
>     wrote:
>     >
>     > Thanks for checking.
>     >
>     > I'd be interested to know if setting a higher compression level
>     for zstd
>     > can get us to a similar compression ratio to xz. If so, then I
>     think it
>     > could be some real value to distro maintainers to be able to *tune*
>     > their compression.
>
>     yeah it will be interesting to say try something like level 9 but
>     I think times
>     might regress with that but it might be good to know the balance
>     and perhaps
>     suggest size mode and performance mode of zstd instead of xz
>

-- 
Alex Stewart
Software Engineer - NI Real-Time OS
NI (National Instruments)

alex.stewart@ni.com