Re: Re: Re: [OE-core] [PATCH] opkg: enable zstd support

[Alex Stewart <alex.stewart@ni.com>]

Thanks for checking.

I'd be interested to know if setting a higher compression level for zstd 
can get us to a similar compression ratio to xz. If so, then I think it 
could be some real value to distro maintainers to be able to *tune* 
their compression.

That's not blocking for your new PR though.


On 9/14/22 05:08, Etienne Cordonnier wrote:
> Also note that zstd's default compression level is 3 per default (from 
> a 1 to 19 scale). I did not test other compression levels.
>
> On Wed, Sep 14, 2022 at 11:58 AM Etienne Cordonnier 
> <ecordonnier@snapchat.com> wrote:
>
>     I ran a build of core-image-full-cmdline using xz and zstd, using
>     pre-populated downloads and sstate-cache directories but with
>     empty tmp directory. Here are the numbers:
>     zstd:
>     bitbake core-image-full-cmdline took 2m52.768s (real), the
>     resulting directory tmp/deploy/ipk is 1.6GB big.
>     xz:
>     bitbake core-image-full-cmdline took 4m14.214s (real), the
>     resulting directory tmp/deploy/ipk/ is 996M big
>
>     So the build with xz is 47% slower (254/172) than with zstd for
>     this "incremental build" use-case.
>
>     See the 5 biggest packages, the difference in compression-ratio
>     increases with big files:
>     build$ ls -lh tmp-zstd/deploy/ipk/core2-64/ --sort=size | head -n5
>     total 1.6G
>     -rw-r--r-- 3 ecordonnier ecordonnier  331M Sep 14 10:39
>     gcc-dbg_12.2.0-r0_core2-64.ipk
>     -rw-r--r-- 3 ecordonnier ecordonnier  264M Sep 14 10:39
>     openssl-dbg_3.0.5-r0_core2-64.ipk
>     -rw-r--r-- 3 ecordonnier ecordonnier  113M Sep 14 10:39
>     openssl-ptest_3.0.5-r0_core2-64.ipk
>     -rw-r--r-- 3 ecordonnier ecordonnier   47M Sep 14 10:39
>     binutils-dbg_2.39-r0_core2-64.ipk
>     build$ ls -lh tmp-xz/deploy/ipk/core2-64/ --sort=size | head -n5
>     total 963M
>     -rw-r--r-- 3 ecordonnier ecordonnier  214M Sep 14 10:44
>     gcc-dbg_12.2.0-r0_core2-64.ipk
>     -rw-r--r-- 3 ecordonnier ecordonnier  193M Sep 14 10:44
>     openssl-dbg_3.0.5-r0_core2-64.ipk
>     -rw-r--r-- 3 ecordonnier ecordonnier   35M Sep 14 10:44
>     openssl-ptest_3.0.5-r0_core2-64.ipk
>     -rw-r--r-- 3 ecordonnier ecordonnier   32M Sep 14 10:44
>     binutils-dbg_2.39-r0_core2-64.ipk
>     ...
>
>     I think for use-cases where the size of the ipk packages matters,
>     xz is better. For use-cases where it does not matter (ipk packages
>     not deployed), build-time matters more than compression-ratio and
>     zstd is better.
>
>     Regarding the enablement of zstd in opkg per default, I'll send a
>     new version of the patch without this line.
>     My thinking for enabling zstd per default in opkg was that zstd is
>     already enabled per default in libarchive's PACKAGECONFIG, and
>     disabling zstd in opkg's PACKAGECONFIG removes only a few lines of
>     code from opkg (opkg uses libarchive for doing the actual
>     compression/decompression).
>
>     On Tue, Sep 13, 2022 at 11:57 PM Alex Stewart
>     <alex.stewart@ni.com> wrote:
>
>
>
>         On 9/13/22 15:20, Alex Feinman wrote:
>         > I do have some numbers. When I was selling this change
>         internally, I
>         > did a comparison on our internal build.
>         >            Combined write IPK times (Σ t do_package_write_ipk)
>         > xz         162m 35s
>         > gz         52m 13s
>         > zstd       33m 49s
>         > Compression rate for zstd was closer to xz than to gz but
>         not as good
>         > as xz. For systems that have to cache packages on the device
>         with
>         > limited storage xz might be a better option, but for the
>         bulk of
>         > projects zstd is the best choice
>         > Additionally, zstd offers much faster decompression than xz
>         so the
>         > rootfs build step that includes unpacking all of the ipks,
>         takes 3m
>         > 58s with xz and 2m 44s with zstd.
>         > One other thing of note - if your build includes debug
>         packages, some
>         > may be quite large. E.g. one of our components produces a
>         2.2 GB debug
>         > package (uncompressed). On large files xz requires a
>         disproportionally
>         > large amount of time resulting in 15 minutes needed to
>         simply write
>         > ipk for the abovementioned packages, whereas zstd took about
>         45 sec.
>         > For frequent tasks like bitbaking a single package this
>         translates in
>         > a lot of saved time.
>
>         Those are certainly compelling performance improvements.
>         Assuming that
>         the final data-segment size is within 5%-ish of xz, then I
>         would agree
>         with the rest of the thread that it should probably be the
>         contemporary
>         default.
>
>         And if we make it the default compressor for OE IPKs, then
>         obviously my
>         criticism in the original PR is satisfied.
>
>         > Bottom line - I think making xz a default package compressor
>         was not
>         > entirely thought through. gzip or zstd is what the default
>         should be.
>
>         ZStandard support was only added to opkg last September [1].
>         Before
>         that, xz was the new hotness that replaced gzip. :)
>
>         [1]
>         https://urldefense.proofpoint.com/v2/url?u=https-3A__git.yoctoproject.org_opkg_commit_-3Fid-3D5dead419e94bce2e6b743ad786c1daec0e1aa294&d=DwIDaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=AhkbNonVuMIGRfPx_Qj9TsyDLWdbBqarUzFxz3aALck&m=VG_fgpCGq8Zgu73KQP1wTtb1D1TZftNpXj8jGnouPtGIBYyLIZbG8F-85POUcVN7&s=Mwoq2kkjfZhto6J5OomduJ5Rhyg_oSe-dkjeltE4Ls8&e=
>         <https://urldefense.proofpoint.com/v2/url?u=https-3A__git.yoctoproject.org_opkg_commit_-3Fid-3D5dead419e94bce2e6b743ad786c1daec0e1aa294&d=DwIDaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=AhkbNonVuMIGRfPx_Qj9TsyDLWdbBqarUzFxz3aALck&m=VG_fgpCGq8Zgu73KQP1wTtb1D1TZftNpXj8jGnouPtGIBYyLIZbG8F-85POUcVN7&s=Mwoq2kkjfZhto6J5OomduJ5Rhyg_oSe-dkjeltE4Ls8&e=>
>
>
>         > One final note: I could not find a reasonable explanation
>         for why
>         > opkg-tools require code changes to support a different
>         compressor. BSD
>         > tar and GNU tar both can easily accept compressors that they
>         have no
>         > idea about (via -I option) because all of them provide a
>         unified
>         > command line interface for use in pipes. If this were done
>         similar to
>         > tar, we could have used any compressor we wanted, including the
>         > multithreaded versions (zstdmt)
>
>         Well, presumably IPK creation tools can only support the
>         matrix of
>         compression algorithms which your opkg binary can decompress.
>         I suppose
>         someone could try to implement a plugable compression module
>         system for
>         opkg. But given that nearly everyone uses opkg in an embedded
>         context,
>         I'm not sure it would get much use.
>
>         >
>         > On Tue, Sep 13, 2022 at 12:43 PM Khem Raj
>         <raj.khem@gmail.com> wrote:
>         >
>         >     On Tue, Sep 13, 2022 at 12:19 PM Alex Stewart
>         >     <alex.stewart@ni.com> wrote:
>         >     >
>         >     > ACK from me - apart from enabling zstd by default.
>         >     >
>         >     > On 9/13/22 07:37, Etienne Cordonnier via
>         lists.openembedded.org
>         <https://urldefense.com/v3/__http://lists.openembedded.org__;!!FbZ0ZwI3Qg!t0JG8eQqpsPs8clofb2phwJc6ONCKBjy-iQukKenLRVzBQvJrKTp8VsgKBI7cR8KlV3N64C3fKRHIcFhOF1_vg$>
>         >   
>          <https://urldefense.com/v3/__http://lists.openembedded.org__;!!FbZ0ZwI3Qg!u0KgIth5yNSO1tok49C5_xo0QPHIFLa4kZBl3vVZwB-2Ui1uYBW7nt85fKXXM_VAHZ2LdFx_bxrNjvo$>
>         >     wrote:
>         >     > > This allows the use of zstd for opkg packages by using
>         >     OPKGBUILDCMD:
>         >     > > OPKGBUILDCMD = "opkg-build -Z zstd"
>         >     > >
>         >     > > Signed-off-by: Alex Feinman <afeinman@snap.com>
>         >     > > Signed-off-by: Etienne Cordonnier <ecordonnier@snap.com>
>         >     > > ---
>         >     > >   meta/recipes-devtools/opkg/opkg_0.6.0.bb
>         <https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!t0JG8eQqpsPs8clofb2phwJc6ONCKBjy-iQukKenLRVzBQvJrKTp8VsgKBI7cR8KlV3N64C3fKRHIcFFmbls2A$>
>         >   
>          <https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!u0KgIth5yNSO1tok49C5_xo0QPHIFLa4kZBl3vVZwB-2Ui1uYBW7nt85fKXXM_VAHZ2LdFx_-65pBFo$>
>         >     | 3 ++-
>         >     > >   1 file changed, 2 insertions(+), 1 deletion(-)
>         >     > >
>         >     > > diff --git
>         a/meta/recipes-devtools/opkg/opkg_0.6.0.bb
>         <https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!t0JG8eQqpsPs8clofb2phwJc6ONCKBjy-iQukKenLRVzBQvJrKTp8VsgKBI7cR8KlV3N64C3fKRHIcFFmbls2A$>
>         >   
>          <https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!u0KgIth5yNSO1tok49C5_xo0QPHIFLa4kZBl3vVZwB-2Ui1uYBW7nt85fKXXM_VAHZ2LdFx_-65pBFo$>
>         >     b/meta/recipes-devtools/opkg/opkg_0.6.0.bb
>         <https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!t0JG8eQqpsPs8clofb2phwJc6ONCKBjy-iQukKenLRVzBQvJrKTp8VsgKBI7cR8KlV3N64C3fKRHIcFFmbls2A$>
>         >   
>          <https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!u0KgIth5yNSO1tok49C5_xo0QPHIFLa4kZBl3vVZwB-2Ui1uYBW7nt85fKXXM_VAHZ2LdFx_-65pBFo$>
>         >     > > index 7b351e8123..e38d9d6f3f 100644
>         >     > > --- a/meta/recipes-devtools/opkg/opkg_0.6.0.bb
>         <https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!t0JG8eQqpsPs8clofb2phwJc6ONCKBjy-iQukKenLRVzBQvJrKTp8VsgKBI7cR8KlV3N64C3fKRHIcFFmbls2A$>
>         >   
>          <https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!u0KgIth5yNSO1tok49C5_xo0QPHIFLa4kZBl3vVZwB-2Ui1uYBW7nt85fKXXM_VAHZ2LdFx_-65pBFo$>
>         >     > > +++ b/meta/recipes-devtools/opkg/opkg_0.6.0.bb
>         <https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!t0JG8eQqpsPs8clofb2phwJc6ONCKBjy-iQukKenLRVzBQvJrKTp8VsgKBI7cR8KlV3N64C3fKRHIcFFmbls2A$>
>         >   
>          <https://urldefense.com/v3/__http://opkg_0.6.0.bb__;!!FbZ0ZwI3Qg!u0KgIth5yNSO1tok49C5_xo0QPHIFLa4kZBl3vVZwB-2Ui1uYBW7nt85fKXXM_VAHZ2LdFx_-65pBFo$>
>         >     > > @@ -30,7 +30,7 @@ inherit autotools pkgconfig ptest
>         >     > >   target_localstatedir := "${localstatedir}"
>         >     > >   OPKGLIBDIR ??= "${target_localstatedir}/lib"
>         >     > >
>         >     > > -PACKAGECONFIG ??= "libsolv"
>         >     > > +PACKAGECONFIG ??= "libsolv zstd"
>         >     >
>         >     > Building in zstd support by default is a little
>         suspect to me.
>         >     >
>         >     > Unless I'm mistaken, OE-core will only build
>         xz-compressed IPKs by
>         >     > default. So zstd support would be unnecessary for a distro
>         >     integrator
>         >     > who just uses upstream OE-core.
>         >     >
>         >     > For distros which use zstd compression in their
>         packages, I think it
>         >     > would be more appropriate to overwrite the opkg
>         PACKAGECONFIG in a
>         >     > .bbappend.
>         >     >
>         >
>         >     This is perhaps fine. I do wonder if there is some
>         performance
>         >     comparison data between xz and zstd compressed ipks
>         >     with opkg, it might help users on making this choice and
>         also if we
>         >     should consider using
>         >     zstd by default at some point or not.
>         >
>         >     > Is there something I'm not considering here?
>         >     >
>         >     > >
>         >     > >   PACKAGECONFIG[gpg] = "--enable-gpg,--disable-gpg,\
>         >     > >       gnupg gpgme libgpg-error,\
>         >     > > @@ -39,6 +39,7 @@ PACKAGECONFIG[gpg] =
>         >     "--enable-gpg,--disable-gpg,\
>         >     > >   PACKAGECONFIG[curl] =
>         "--enable-curl,--disable-curl,curl"
>         >     > >   PACKAGECONFIG[ssl-curl] =
>         >     "--enable-ssl-curl,--disable-ssl-curl,curl openssl"
>         >     > >   PACKAGECONFIG[sha256] =
>         "--enable-sha256,--disable-sha256"
>         >     > > +PACKAGECONFIG[zstd] =
>         "--enable-zstd,--disable-zstd,zstd"
>         >     > >   PACKAGECONFIG[libsolv] =
>         >     "--with-libsolv,--without-libsolv,libsolv"
>         >     > >
>         >     > >   EXTRA_OECONF:class-native =
>         >  "--localstatedir=/${@os.path.relpath('${localstatedir}',
>         >     '${STAGING_DIR_NATIVE}')}
>         >     --sysconfdir=/${@os.path.relpath('${sysconfdir}',
>         >     '${STAGING_DIR_NATIVE}')}"
>         >     > >
>         >     > >
>         >     > >
>         >     >
>         >     > --
>         >     > Alex Stewart
>         >     > Software Engineer - NI Real-Time OS
>         >     > NI (National Instruments)
>         >     >
>         >     > alex.stewart@ni.com
>         >     >
>         >     >
>         >     > -=-=-=-=-=-=-=-=-=-=-=-
>         >     > Links: You receive all messages sent to this group.
>         >     > View/Reply Online (#170608):
>         >
>         https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_g_openembedded-2Dcore_message_170608&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=_VWADuvDS3QdSEYWTnh8wNH0lOxVHty18QZynvOtpcU&e=
>         <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_g_openembedded-2Dcore_message_170608&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=_VWADuvDS3QdSEYWTnh8wNH0lOxVHty18QZynvOtpcU&e=>
>         >   
>          <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_g_openembedded-2Dcore_message_170608&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=_VWADuvDS3QdSEYWTnh8wNH0lOxVHty18QZynvOtpcU&e=
>         <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_g_openembedded-2Dcore_message_170608&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=_VWADuvDS3QdSEYWTnh8wNH0lOxVHty18QZynvOtpcU&e=>>
>         >
>         >     > Mute This Topic:
>         >
>         https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_mt_93654146_1997914&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=QncdL2I_pCDsz7AMbsTgStzC7bt5V2x_E_gIblrtjZY&e=
>         <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_mt_93654146_1997914&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=QncdL2I_pCDsz7AMbsTgStzC7bt5V2x_E_gIblrtjZY&e=>
>         >   
>          <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_mt_93654146_1997914&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=QncdL2I_pCDsz7AMbsTgStzC7bt5V2x_E_gIblrtjZY&e=
>         <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_mt_93654146_1997914&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=QncdL2I_pCDsz7AMbsTgStzC7bt5V2x_E_gIblrtjZY&e=>>
>         >
>         >     > Group Owner:
>         openembedded-core+owner@lists.openembedded.org
>         <mailto:openembedded-core%2Bowner@lists.openembedded.org>
>         >     <mailto:openembedded-core%2Bowner@lists.openembedded.org
>         <mailto:openembedded-core%252Bowner@lists.openembedded.org>>
>         >     > Unsubscribe:
>         >
>         https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_g_openembedded-2Dcore_unsub&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=PxAE_3yGzbSZ21-LHz_1r3gd2_STPoZcrYDeu9vd_GM&e=
>         <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_g_openembedded-2Dcore_unsub&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=PxAE_3yGzbSZ21-LHz_1r3gd2_STPoZcrYDeu9vd_GM&e=>
>         >   
>          <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_g_openembedded-2Dcore_unsub&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=PxAE_3yGzbSZ21-LHz_1r3gd2_STPoZcrYDeu9vd_GM&e=
>         <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.openembedded.org_g_openembedded-2Dcore_unsub&d=DwIBaQ&c=ncDTmphkJTvjIDPh0hpF_4vCHvabgGkICC2epckfdiw&r=ixV5wgZimxpcSCog35wonZKeFaeEVKUbESWNj_K-1j0&m=KvCRyrKojziBs0h9aEjEfI4R_J4FnZIK8nDBqKo_I1CpnrIJ78DBxgMrvY25Gwyh&s=PxAE_3yGzbSZ21-LHz_1r3gd2_STPoZcrYDeu9vd_GM&e=>>
>         >      [raj.khem@gmail.com]
>         >     > -=-=-=-=-=-=-=-=-=-=-=-
>         >     >
>         >
>
>         -- 
>         Alex Stewart
>         Software Engineer - NI Real-Time OS
>         NI (National Instruments)
>
>         alex.stewart@ni.com
>

-- 
Alex Stewart
Software Engineer - NI Real-Time OS
NI (National Instruments)

alex.stewart@ni.com