Space optimizations in kirkstone and scarthgap for sstate objects

Space optimizations in kirkstone and scarthgap for sstate objects

[Richard Purdie]

We've been trying to resolve some of our sstate CDN issues and one
thing causing us big headaches are large sstate files. By far the worst
offenders are webkitgtk and llvm.

In master (and in styhead) we have these:

https://git.yoctoproject.org/poky/commit/?id=d9066258a1e48e75cf814fb0940ac420bcdd6e99
https://git.yoctoproject.org/poky/commit/?id=aca8acceb87e568115da9696732e915d91da9652
https://git.yoctoproject.org/poky/commit/?id=1d8d9b1f87b21af36d9c328132f71c528eefd5b0

which did significantly reduce the size of the output. Unfortunately I
made the first two work with changes to DEBUG_LEVELFLAG which are not
suited for backporting:

https://git.yoctoproject.org/poky/commit/?id=1cf0974ad242f7eb2815a4ef0e3e5b6507ca56ea
https://git.yoctoproject.org/poky/commit/?id=ae4ec59b3eff85aaa1cfe914a134e56e1d125744

I do think we should apply a similar tweak to kirkstone and scarthgap
though. The third patch above shouldbackport. We should be able to do
something equivalent for llvm. Any thoughts?

Michael obtained some sstate size info which I've shared below. You can
tell the age by the sstate version number (10, 12 and 14 are kirkstone,
scarthgap and styhead/master) and the recipe versions.

Cheers,

Richard

104M	./00/ed/sstate:lib32-llvm:x86-pokymllib32-linux:19.1.3:r0:x86:14:00ed0ece3a83078a289a4437c8e910ea039367fc6557429e87c3edbb7b3d1e12_package.tar.zst
104M	./20/35/sstate:lib32-llvm:x86-pokymllib32-linux:19.1.2:r0:x86:14:2035ee495ab5f8882a1c67a61de3dbd44b5f9ee0c1c060e195a7066d3eac7054_package.tar.zst
104M	./2b/70/sstate:lib32-llvm:x86-pokymllib32-linux:19.1.3:r0:x86:14:2b70d3131e8c75727fc7874f08bdb583952a78163acb80b9d456e39887a54736_package.tar.zst
104M	./58/f2/sstate:lib32-llvm:x86-pokymllib32-linux:19.1.2:r0:x86:14:58f27abfcf6e3ef566cc88bd41a1a961d0ce75cdb9c8544f546e4b9d91e7af0a_package.tar.zst
104M	./5f/ba/sstate:lib32-llvm:x86-pokymllib32-linux:19.1.2:r0:x86:14:5fbacd6f74e3f9929171d6f86f5a3b2bb81e139b5e1a546e153c0998048ecb88_package.tar.zst
104M	./91/9f/sstate:lib32-llvm:x86-pokymllib32-linux:19.1.3:r0:x86:14:919f40ad5f7011e2ee452736853aeaf4d00c8327d4fa8490796c92a4754ca35d_package.tar.zst
104M	./ae/49/sstate:lib32-llvm:x86-pokymllib32-linux:18.1.8:r0:x86:14:ae499c9617c5e521f4f46e5d03a57521a609079221b28b1fa3332963456d88f2_package.tar.zst
104M	./af/81/sstate:lib32-llvm:x86-pokymllib32-linux:19.1.2:r0:x86:14:af819e748f3353d856f7337d431c1ac0d24f322d9e48dca3f8bd5ead8fc07ac1_package.tar.zst
104M	./c0/50/sstate:lib32-llvm:x86-pokymllib32-linux:18.1.8:r0:x86:14:c0506a6fff94998b2a40cdd322b0888ee5c1dbdcb906957d011fc8e47d63e873_package.tar.zst
104M	./d3/4c/sstate:lib32-llvm:x86-pokymllib32-linux:19.1.2:r0:x86:14:d34c7e0ea79381bba76ce0942ba01a791dfe3b1e5351ae4a78c7193c47a93bbb_package.tar.zst
104M	./e6/5e/sstate:lib32-llvm:x86-pokymllib32-linux:19.1.3:r0:x86:14:e65ef0c6639d9d75a103044b501a3c91d48253694235746cca1261b09ae1e3fe_package.tar.zst
104M	./f7/2b/sstate:lib32-llvm:x86-pokymllib32-linux:19.1.2:r0:x86:14:f72b8c32c418df5d1f699eb9fbab914124efa50c8becd7ec3ee819d33bfa1367_package.tar.zst
2.8G	./19/48/sstate:lib32-llvm:x86-pokymllib32-linux:18.1.6:r0:x86:12:1948cd3c34fe2edd01ad0b056dcbff38c69402dd2a665e598f95a0c7c1b50a4a_package.tar.zst
2.8G	./3c/75/sstate:lib32-llvm:x86-pokymllib32-linux:18.1.6:r0:x86:12:3c755ce9707342753797460c588b1485d2c15c997bf0e4a34ec7183954bbef0e_package.tar.zst

141M	./20/ef/sstate:webkitgtk:core2-32-poky-linux-musl:2.44.3:r0:core2-32:14:20efb85cd3c0ee34a9cc88c6473916016f9d8185b5d2025f417eb635401c4a4a_package_write_deb.tar.zst
141M	./25/78/sstate:webkitgtk:core2-32-poky-linux-musl:2.44.3:r0:core2-32:14:257831f0d6e12d85cdae719f2b2f872b20088b9a26dbd2baa0cd7b90a72b5eff_package_write_deb.tar.zst
141M	./3e/4d/sstate:webkitgtk:core2-32-poky-linux-musl:2.44.3:r0:core2-32:14:3e4d2f88049897c9bdb7e4fdac37dec97044d9ad71857ebb272bedd3f736273c_package_write_deb.tar.zst
141M	./6f/15/sstate:webkitgtk:core2-32-poky-linux-musl:2.44.3:r0:core2-32:14:6f15085640878bcbe768c8f23500b7b8a095f91ad666106c00492fef714c0b74_package_write_deb.tar.zst
141M	./73/45/sstate:webkitgtk:core2-32-poky-linux-musl:2.44.3:r0:core2-32:14:7345ad746eb2df017467c22b611e69a66dfd392da387e2bc2c43148dc1a33c78_package_write_deb.tar.zst
141M	./81/b0/sstate:webkitgtk:core2-32-poky-linux-musl:2.44.3:r0:core2-32:14:81b03b15b536d1d79888a16976caf54558623c6cb300165f6371037d48670eaf_package_write_deb.tar.zst
141M	./81/e0/sstate:webkitgtk:core2-32-poky-linux-musl:2.44.3:r0:core2-32:14:81e0aca91f089edbfdf4ed2c6fd237005a947e521c868225e194db16c8d7a00d_package_write_deb.tar.zst
141M	./98/35/sstate:webkitgtk:core2-32-poky-linux-musl:2.44.3:r0:core2-32:14:9835757409db010ebd124f25f71b4306dcd5de1e4b1b50b6f56b00a9aa790370_package_write_deb.tar.zst
141M	./9e/6f/sstate:webkitgtk:core2-32-poky-linux-musl:2.44.3:r0:core2-32:14:9e6f9b37902da55ca115641dd5324d9611583fef25301033b44832ce80980d56_package_write_deb.tar.zst
141M	./a7/c4/sstate:webkitgtk:core2-32-poky-linux-musl:2.44.3:r0:core2-32:14:a7c4a43aabab0c83192c574b10e3423a48f2d6c27b314d2787afa6a68435b329_package_write_deb.tar.zst
141M	./d0/7f/sstate:webkitgtk:core2-32-poky-linux-musl:2.44.3:r0:core2-32:14:d07f98cdec4280efef71a74eb7affbfc12401cd5400847a96eb88ea1f2a608be_package_write_deb.tar.zst
141M	./d4/09/sstate:webkitgtk:core2-32-poky-linux-musl:2.44.3:r0:core2-32:14:d40901211cfc9ce41d3bf776216d88577762a63588dd055cc5b7a82424c26c21_package_write_deb.tar.zst
141M	./de/53/sstate:webkitgtk:core2-32-poky-linux-musl:2.44.3:r0:core2-32:14:de536adfbcb7f628899f85e4d67ebd177a28bc4cf8215fe3575a5b4430190434_package_write_deb.tar.zst
141M	./df/fd/sstate:webkitgtk:core2-32-poky-linux-musl:2.44.3:r0:core2-32:14:dffd189177bc9d3d70ba4a7b92bbe1b0a40d0aa06ab3d7fb953a460b0b52e1d5_package_write_deb.tar.zst
141M	./f9/30/sstate:webkitgtk:core2-32-poky-linux-musl:2.44.3:r0:core2-32:14:f930423d392a2e72ba229901c9866c8da835e04ef29222886499aa81fd5c5d2d_package_write_deb.tar.zst
149M	./04/16/sstate:webkitgtk:core2-32-poky-linux-musl:2.44.3:r0:core2-32:14:04165a096dec995e70af61cd4a3f09640f5d97c2143232e88bf0d98e0e823e3a_package_write_rpm.tar.zst
149M	./09/b4/sstate:webkitgtk:core2-32-poky-linux-musl:2.44.3:r0:core2-32:14:09b4b9e3198f07baf42851a6b26e8b03c4302378b64233bc70e022e1b53a0d33_package_write_rpm.tar.zst
149M	./0d/9a/sstate:webkitgtk:core2-32-poky-linux-musl:2.44.3:r0:core2-32:14:0d9ad17b8a5f239ed771fcbde4bbc3b6e473c4621c1985b014983471a4845ba9_package_write_rpm.tar.zst
149M	./1c/74/sstate:webkitgtk:core2-32-poky-linux-musl:2.44.3:r0:core2-32:14:1c74b8f12e9fdcea710171e59ba1818293bd17827ccca4c5d1b190f5642809c3_package_write_rpm.tar.zst
149M	./26/0b/sstate:webkitgtk:core2-32-poky-linux-musl:2.44.3:r0:core2-32:14:260b4b8a2119db7b80d4aaabb2d444913bb523f5c9c8dbc3276ebd668949a21e_package_write_rpm.tar.zst
149M	./52/f8/sstate:webkitgtk:core2-32-poky-linux-musl:2.44.3:r0:core2-32:14:52f85fd66d3221c35c28601c84ff4aaf59a368b70fee6e03596da98d64ed6c3d_package_write_rpm.tar.zst
149M	./5a/2b/sstate:webkitgtk:core2-32-poky-linux-musl:2.44.3:r0:core2-32:14:5a2bd9bf54aa8b3f6acd8e84fe93f66107c7d189c000e7da9d1507912c625d3d_package_write_rpm.tar.zst
149M	./7c/ce/sstate:webkitgtk:core2-32-poky-linux-musl:2.44.3:r0:core2-32:14:7cce9dc1476f0f09be20f3dd1f8ec55837fd21db476f433aee5b4e9d770274fd_package_write_rpm.tar.zst
149M	./88/df/sstate:webkitgtk:core2-32-poky-linux-musl:2.44.3:r0:core2-32:14:88dfc8108f84cb336897d6b94a626bbb0720deed46ddfa628b59e3273c5314ff_package_write_rpm.tar.zst
149M	./a0/50/sstate:webkitgtk:core2-32-poky-linux-musl:2.44.3:r0:core2-32:14:a05037502d9776bd9b933440baf22de6f375e418c584ead15259621a3164de65_package_write_rpm.tar.zst
149M	./b4/43/sstate:webkitgtk:core2-32-poky-linux-musl:2.44.3:r0:core2-32:14:b443ec3e71b2c8e17710d72469bf4d772f745d1e60c175597fcf2492dbe4d2b8_package_write_rpm.tar.zst
149M	./d1/ec/sstate:webkitgtk:core2-32-poky-linux-musl:2.44.3:r0:core2-32:14:d1ec7f8800202e6fba41575a601a2d603b984d4e1548ee2415626874528f425f_package_write_rpm.tar.zst
149M	./f5/a0/sstate:webkitgtk:core2-32-poky-linux-musl:2.44.3:r0:core2-32:14:f5a00fe83ada7b8c29e91866d592794b494b3051bed87058f00a3b6c7bd0eece_package_write_rpm.tar.zst
149M	./fa/a9/sstate:webkitgtk:core2-32-poky-linux-musl:2.44.3:r0:core2-32:14:faa91a6aa1d8bd916d1fd01b3dd8e379757ac82aa147a674d74002d13660995b_package_write_rpm.tar.zst
161M	./0e/b8/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:0eb81f641b8f4d11b2ff582b99131ae318a8f150c5aaa74091a7f3e8a8cfd0f8_package_write_deb.tar.zst
161M	./0f/fc/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:0ffcc71c5a37debe0b0c891d2bb52f1b7d0bfb905c738db14b9d4c71028e500b_package_write_deb.tar.zst
161M	./10/f6/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:10f65b395c396b4ba6afaec58e59e1a9f392ef995b55444449f66b6e796e5335_package_write_deb.tar.zst
161M	./18/b8/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:18b888fa0047f51562f7224a57480b85edfab3fcda3f5cf1a2c44226e0bc23e7_package_write_deb.tar.zst
161M	./1e/44/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:1e44a9194baab94768a4155f2f89da42f2452fb4bd08e4ef8d778e4a959bc61f_package_write_deb.tar.zst
161M	./24/b4/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:24b44f81c2ece75607a33822a1e86f70a4f1819ffbb89526a07d20cc94074227_package_write_deb.tar.zst
161M	./26/03/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:26033539ffe92ec0bdb59b3390c71e0689340d080c17c47192e5e1596dcc83c7_package_write_deb.tar.zst
161M	./45/ce/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:45ce47ced5268685608e36ce5c986f0ebc22e9bacbf47e3d6a6a79ab14cfbc97_package_write_deb.tar.zst
161M	./48/c0/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:48c0cc29042b0116118d3326d461c78986d2a1d1f445be05bcc0be3cabf2ba8e_package_write_deb.tar.zst
161M	./4e/42/sstate:webkitgtk:core2-64-poky-linux-musl:2.44.3:r0:core2-64:14:4e42645d6f963059448b97918f886a0005dada8c2fd880ec197072daef61f681_package_write_deb.tar.zst
161M	./53/c0/sstate:webkitgtk:core2-64-poky-linux-musl:2.44.3:r0:core2-64:14:53c01c1ef706fdaff411aadd0fce7d643fea1d53aed1abf62a8965c0af3b997c_package_write_deb.tar.zst
161M	./68/e6/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:68e6f28962041dd48372ca658d81ace0b45cc2f46f1c023f8893996759ef7167_package_write_deb.tar.zst
161M	./72/aa/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:72aa5f82f56da733109d9ccfc5c7117d1c59503758c4825a9b920d8080f12154_package_write_deb.tar.zst
161M	./81/5d/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:815d6afe058fe588e6cc6d19cb0849f3616ed4d91d50c01afdd86ac2fe92aec0_package_write_deb.tar.zst
161M	./85/60/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:8560787196254f05ae712dc5b1c900fc74f30aebc2aa692b74044ebd15601070_package_write_deb.tar.zst
161M	./85/c4/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:85c426548be7f40833caee34baa7b7d34a9218a6062f883ba142aea5380338f0_package_write_deb.tar.zst
161M	./8e/8e/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:8e8e632aa63519ad1cac430c52b22696ec2719691c10fb7c4175236d0d269c69_package_write_deb.tar.zst
161M	./97/f2/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:97f20034f5f1dae1b0ddbb2fd83cb8854fb7d52ff0b85626b7ab6234a73b1634_package_write_deb.tar.zst
161M	./99/0e/sstate:webkitgtk:core2-64-poky-linux-musl:2.44.3:r0:core2-64:14:990efa08f9a5e24e1b3aa1218acd3123b5802f2b801b0669d87bdf15f4371a90_package_write_deb.tar.zst
161M	./9a/20/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:9a2057d33db6b77c613c37012f19a11976142478ef424aad319615fa05621e12_package_write_deb.tar.zst
161M	./9a/43/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:9a436c73e987e281eba5bafb73944e12eb5d1fba3c85cab902ffe9ca3bd48cdc_package_write_deb.tar.zst
161M	./9a/f6/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:9af66eab660f5f68c726a134ade09a0db76f056cb8656813700cdd2125479e0a_package_write_deb.tar.zst
161M	./9d/0f/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:9d0fda25586df4b1ee8df34217cdfec4400139d72edc4b52048f90daf0da70d5_package_write_deb.tar.zst
161M	./9d/a4/sstate:webkitgtk:core2-64-poky-linux-musl:2.44.3:r0:core2-64:14:9da4fb869d2ed831d0069409aa992e2780c2d3547defc4e844b838e4c25b99b2_package_write_deb.tar.zst
161M	./9e/a0/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:9ea0aa73939833221956c3606ebf72b95ca5e32bc50cb224f10b453b881867d0_package_write_deb.tar.zst
161M	./a0/12/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:a0126661cd631fde1f0b4343f88b8f65c0323a9f42768da126f22a272a07ed21_package_write_deb.tar.zst
161M	./a0/95/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:a095e51b472c32ccaab1c94b7df272952899ad49089b2e382f4488bf5a994fcc_package_write_deb.tar.zst
161M	./bb/02/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:bb022f4e4d2dbdd4ba16e7ffbed0b87295558f63f827effa23d1c8d5f1bc865e_package_write_deb.tar.zst
161M	./be/a6/sstate:webkitgtk:core2-64-poky-linux-musl:2.44.3:r0:core2-64:14:bea6c6df6662f47337068e08f750ce0030ad1e0c5e4eb7fbfc9159a91a8a4c81_package_write_deb.tar.zst
161M	./c0/53/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:c053152469e32b9b64ad016a190dde64c92af2b8e2d6d5b9bfb6c8195ac93ab5_package_write_deb.tar.zst
161M	./c6/27/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:c62781ec9137692284bf3ea9ae074e339693ff604d0402463cd7c47a5b133a47_package_write_deb.tar.zst
161M	./c8/21/sstate:webkitgtk:core2-64-poky-linux-musl:2.44.3:r0:core2-64:14:c821c9d3842a27cb4293d723a1564fd6c1c276a9acbc302173dec01ae7cfc89f_package_write_deb.tar.zst
161M	./d9/c8/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:d9c886fdc16678b919aa505b9cf832de31439164521453cc92e5a2b3ada506d3_package_write_deb.tar.zst
161M	./e4/b1/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:e4b1c2dad2d2419858c8767d0ce3b51dc983e704935be781667b4018f6574e8f_package_write_deb.tar.zst
161M	./f4/a0/sstate:webkitgtk:core2-64-poky-linux-musl:2.44.3:r0:core2-64:14:f4a053c0dcbd265d3d6a1ba3d65ea4ec791a8db2b1e6ca17d6125553fd70b804_package_write_deb.tar.zst
161M	./f6/66/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:f6665d850dac513be9fd0dcb1db9872a257ebd9ae25b3c013682beaf273249a4_package_write_deb.tar.zst
161M	./fc/aa/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:fcaa7ff6f0991a0a5c4bcd9235dd78545632380ffc610acdc844df32d7499a49_package_write_deb.tar.zst
161M	./fc/ea/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:fceaf66da44694ff91df3fb3cfe4ea6982b939956d79673a23b51a96322362ec_package_write_deb.tar.zst
161M	./fd/1a/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:fd1a184c12e0b6d17ae0c8979067a1990d29dc4f397592392b4f4d1dde5e89c0_package_write_deb.tar.zst
161M	./ff/e6/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:ffe6955c1019d5f05fdf2d7001bb12f00a284df8326bc01abc14f9f89418657e_package_write_deb.tar.zst
163M	./09/91/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:09914b620c4f735cbc9a3729255c77bd8b22f464a04ddd00424c2b616ccdeb74_package_write_deb.tar.zst
163M	./60/61/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:60611512b1775ceec5d515513d10822da1db4082b09d8309310832be7c9ed986_package_write_deb.tar.zst
163M	./67/7b/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:677bcce40437bd9a89dde55a86c10584e9113a3f4e80ceb50b0a498df28eec40_package_write_deb.tar.zst
163M	./9b/77/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:9b77f2fe37d03f1e54897f6b245dd3fa4cdd6b8f73bad801f009b4c4fa767b80_package_write_deb.tar.zst
163M	./a8/fa/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:a8fa14384fa85d7e48a61d211bcb6816805affebf94f9ffcbe2eac3dd9b4498e_package_write_deb.tar.zst
163M	./a9/03/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:a903c7c9e2d4997b1aa3559e969442a0b01443768e8c8958b1222653c2da0535_package_write_deb.tar.zst
163M	./f1/fb/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:f1fbefb9980b3ef8dd29b7ca97af04afaa160e485230414e65216cf989c1a707_package_write_deb.tar.zst
169M	./13/e8/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:13e848ae06098b63248ddabed59898bdd0dba13d2ef958303248042c68148dfa_package_write_rpm.tar.zst
169M	./1c/6d/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:1c6db04ebe96bf6cb0b4d65d9379013b04de0be832feceaa51f900d06e238fb5_package_write_rpm.tar.zst
169M	./20/d5/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:20d506a7004288a6e7afe5b0ebf8a0a48629efdc404794e3b6efcbc06bbf2388_package_write_rpm.tar.zst
169M	./28/02/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:28028063e900e5c6ebbd00b95253985124e9c2389b2eaf8c847b7b11fd66f9f0_package_write_rpm.tar.zst
169M	./2a/86/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:2a86b6c21d19531d6ab81f6579b938dc1754c7dc1a15e876a637b63b6235af51_package_write_rpm.tar.zst
169M	./41/4c/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:414c964f9038f808ac03c292734a401c6a9e3186781500900487fdce51cf57f7_package_write_rpm.tar.zst
169M	./47/79/sstate:webkitgtk:core2-64-poky-linux-musl:2.44.3:r0:core2-64:14:4779b0ee435468598a8eff1371050655890e077d0fc751f5daa187ac3824a55b_package_write_rpm.tar.zst
169M	./55/86/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:558643bf9eca21f17fe8ecab2bca35dc5263cd8ab58d5c5fdee5e356bdf34de2_package_write_rpm.tar.zst
169M	./6a/1b/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:6a1b7af6131e06213d123ab44ed133d025a903cf9bbe5909f4409ef1a58358e4_package_write_rpm.tar.zst
169M	./6b/7e/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:6b7e1ec5423aba27449fd1d81bed970ebffa85ce623fa33a336b616b0dbae36e_package_write_rpm.tar.zst
169M	./71/19/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:7119df688b9e60f645ed88c160a306fdf4c1bef411e9c123953feb97ebca4b13_package_write_rpm.tar.zst
169M	./76/4d/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:764d40ebedc24be2d1a0c55c9046d694a2a13515485de862723203a242decf71_package_write_rpm.tar.zst
169M	./77/0e/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:770e65c4dd57b69ae25206424d1a021b41ca6a8f4bc5afb5f6a2c116855faa79_package_write_rpm.tar.zst
169M	./7b/7b/sstate:webkitgtk:core2-64-poky-linux-musl:2.44.3:r0:core2-64:14:7b7bc3a0fb286f710e0e854781a0ca3410b4cbf903e98a18dc9311d9e6646b15_package_write_rpm.tar.zst
169M	./7c/50/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:7c502844df000a75d731428d5cf137742c386e1210b58340b19aaca7f11628cf_package_write_rpm.tar.zst
169M	./80/75/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:80750025d64f5e296cb989e5513f871ed42783e552c513719b5b051d8a6db23b_package_write_rpm.tar.zst
169M	./86/83/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:8683920be2f584c1df13545711c9ee67389f6618b27cdbf6248ca4c134406b2f_package_write_rpm.tar.zst
169M	./8c/a7/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:8ca7f8df36a1bb3b94e247e813c8ac5d33ff4f7bf4aa519e786b6b44241e91a1_package_write_rpm.tar.zst
169M	./8f/06/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:8f062a005bd44eecbb871293863329b59c5326342af5481ffea044db2ff4a65b_package_write_rpm.tar.zst
169M	./95/2f/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:952f907b09aee343f6eec25e7ab3e773d93b9079fc6800f2155d6a7de5dd0d48_package_write_rpm.tar.zst
169M	./99/56/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:9956506cc84f0623cc7ea48865b4eb045f3f5c6d084549a1782dc615924dee37_package_write_rpm.tar.zst
169M	./99/74/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:9974234bcc2fcf5902b796ddf9eb6b1cef58360f2710ac5c97e5c68f81ba213a_package_write_rpm.tar.zst
169M	./a1/a0/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:a1a0492100f78078821aead4b76267f18d9c22d96075ce6331276065b98acb2e_package_write_rpm.tar.zst
169M	./a2/b1/sstate:webkitgtk:core2-64-poky-linux-musl:2.44.3:r0:core2-64:14:a2b1e7750b9d4f70be8b40a49d6f4dcf8e247efccf0aa6e88ab1bedeeed5bfdf_package_write_rpm.tar.zst
169M	./af/cf/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:afcfc81a9b18f557529948d45ae3731e0264a462e9ba8aa7740703d44152f7fd_package_write_rpm.tar.zst
169M	./b0/33/sstate:webkitgtk:core2-64-poky-linux-musl:2.44.3:r0:core2-64:14:b0336d91f8fe895a9b3876af1ae744a2095b2bc0383ea240392634d2b44918e8_package_write_rpm.tar.zst
169M	./b1/4d/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:b14dde639142c2e8d32ffdb37d7d3260cf2d651fe8c6fa0045e40b95e3fcefe5_package_write_rpm.tar.zst
169M	./b5/b2/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:b5b265f8f615fb1997ea8cf8a85d47839f4853d3f555d28082caa70eceb00501_package_write_rpm.tar.zst
169M	./b7/9d/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:b79dca407df957c2bade16ffdea8dbb279dd0b5f87e447bea347aa049b0db280_package_write_rpm.tar.zst
169M	./bc/8f/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:bc8f09f105089b77c4177741028804eccb1da9625c09c92b21172aeb4f797a9b_package_write_rpm.tar.zst
169M	./df/5a/sstate:webkitgtk:core2-64-poky-linux-musl:2.44.3:r0:core2-64:14:df5a1bf70e5d056bc098d58da459583ea3af30359ee3c2cccf34db57d762c02c_package_write_rpm.tar.zst
169M	./ee/31/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:ee3128bd24a09d42e1a07b5f3efe1e8f1be67c7b79b633f3443f3eeaea98fc03_package_write_rpm.tar.zst
169M	./ef/49/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:ef499a42ae0b3bbe175207137b9e0281c9b0d084504abca3c2da33d016b9e8e5_package_write_rpm.tar.zst
169M	./f2/86/sstate:webkitgtk:core2-64-poky-linux-musl:2.44.3:r0:core2-64:14:f286e5fa34fcb18992bcc20a9b62d7c74ce4f5490cbd485cc2c06d6fc8dbf026_package_write_rpm.tar.zst
169M	./f4/96/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:f496b212a064a17a1c09bab96245453eecd562a9895b45a3ab5925df34a39a7c_package_write_rpm.tar.zst
171M	./3d/1c/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:3d1c2b44abdb28969867de40cfbb6b39346abed439b8dbd6b03857191887b004_package_write_rpm.tar.zst
171M	./54/ba/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:54baeec58d2b0bfc385991f98d7e687e47119c2d1451f8974430922c75601675_package_write_rpm.tar.zst
171M	./71/c6/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:71c64b9fa1b734a7b0921cca0ec7907f05c63a4f618224e4775b7038dd499f31_package_write_rpm.tar.zst
171M	./b6/7f/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:b67f23b215f0eab081cc7a60cb79852c6543f056991df1d164b510ed935f61f8_package_write_rpm.tar.zst
171M	./ba/89/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:ba8996a19ab152bdd6d6cc16f102123ed4bfd5a5c574b07c68e4c867739d8b02_package_write_rpm.tar.zst
171M	./e0/72/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:e072d64be41fa03b991708e337b75d108d7bf1c32127d7758f9d9d737f79531b_package_write_rpm.tar.zst
203M	./19/bc/sstate:webkitgtk:core2-32-poky-linux-musl:2.44.3:r0:core2-32:14:19bc5d6634d72e11c60ea670c0587e04c84780c76b7d36ba87ca897979080bc4_package_write_ipk.tar.zst
203M	./26/43/sstate:webkitgtk:core2-32-poky-linux-musl:2.44.3:r0:core2-32:14:2643fad4e23d55f33df17ad47f9c0f0eee397bbcc0ef8d1464a42e99a1b1cc00_package_write_ipk.tar.zst
203M	./33/64/sstate:webkitgtk:core2-32-poky-linux-musl:2.44.3:r0:core2-32:14:33640214b05f43a59242256e47e581b89face33aa208ddbad55aa9676a406a01_package_write_ipk.tar.zst
203M	./3c/ea/sstate:webkitgtk:core2-32-poky-linux-musl:2.44.3:r0:core2-32:14:3cea6c35413e8e5830f41272f66ad9085ee9827c40709f1799e4249a4f79adf4_package_write_ipk.tar.zst
203M	./42/13/sstate:webkitgtk:core2-32-poky-linux-musl:2.44.3:r0:core2-32:14:4213e923848f2627f7be1b3e933c63dc5e08340b7ed09021449379d4e5e8f42c_package_write_ipk.tar.zst
203M	./4b/ec/sstate:webkitgtk:core2-32-poky-linux-musl:2.44.3:r0:core2-32:14:4bec338e9ee4876f9204fe1a1622ddb7bc1f3b2351e362d19d443b52ca3e4466_package_write_rpm.tar.zst
203M	./55/6b/sstate:webkitgtk:core2-32-poky-linux-musl:2.44.3:r0:core2-32:14:556b5c441af244e2e90942bf9f7256f203d33bcbc303beb034e5e73a2c9e224f_package_write_ipk.tar.zst
203M	./56/8e/sstate:webkitgtk:core2-32-poky-linux-musl:2.44.3:r0:core2-32:14:568e859c35943c914d09df701bdaf3e6178d78c9417c63b5d9ae7606c28dc32c_package_write_ipk.tar.zst
203M	./62/5c/sstate:webkitgtk:core2-32-poky-linux-musl:2.44.3:r0:core2-32:14:625c113eea6ad7cc60ee271326befb1b81eafbeb72538116dd7e792c449b2f2e_package_write_ipk.tar.zst
203M	./6d/bb/sstate:webkitgtk:core2-32-poky-linux-musl:2.44.3:r0:core2-32:14:6dbbb0ac54c61c40bb4d2f8f649334d0dab7f1714cf7c5b33ab857a6bffee2e4_package_write_ipk.tar.zst
203M	./79/5b/sstate:webkitgtk:core2-32-poky-linux-musl:2.44.3:r0:core2-32:14:795b7669fac492282bbcf4a3ddbbaaea2e523dd0652854bf71473248b2903f42_package_write_ipk.tar.zst
203M	./8c/f7/sstate:webkitgtk:core2-32-poky-linux-musl:2.44.3:r0:core2-32:14:8cf72c732c1c21b11794381323154bfc2953b48ce8bc0f63ea6c5ecaf09a42c3_package_write_rpm.tar.zst
203M	./a0/45/sstate:webkitgtk:core2-32-poky-linux-musl:2.44.3:r0:core2-32:14:a045dd3051b757a78ebb98528314e2cfad725784bf113cd4000456730d4f26fa_package_write_ipk.tar.zst
203M	./a4/16/sstate:webkitgtk:core2-32-poky-linux-musl:2.44.3:r0:core2-32:14:a4161cd61073bfa56b4d92548988053fe0cce5b256c5ab932a3f883a53be612a_package_write_rpm.tar.zst
203M	./b3/d8/sstate:webkitgtk:core2-32-poky-linux-musl:2.44.3:r0:core2-32:14:b3d8a637e51a1002c6a8b1bc485a450c4d82a63f7d98362810eee6831e65279a_package_write_ipk.tar.zst
203M	./d4/48/sstate:webkitgtk:core2-32-poky-linux-musl:2.44.3:r0:core2-32:14:d448336cc443778e7ffccdd3d93a9bd1763c7577329edce1506a91e6755206d4_package_write_ipk.tar.zst
203M	./ed/5c/sstate:webkitgtk:core2-32-poky-linux-musl:2.44.3:r0:core2-32:14:ed5c64e54010a39f41025a03afe6c8c7a36f0dc055305640b74c08a5c0864fd9_package_write_ipk.tar.zst
203M	./f7/06/sstate:webkitgtk:core2-32-poky-linux-musl:2.44.3:r0:core2-32:14:f706659727cd1b65cc220576b9af79f438d6c2663e8ebffef1004792a54c897b_package_write_ipk.tar.zst
226M	./00/43/sstate:webkitgtk:core2-64-poky-linux-musl:2.44.3:r0:core2-64:14:0043ec0ba41f0aa648b574731360fc82b400141f0db6c556e47ba054f0e45adf_package_write_ipk.tar.zst
226M	./03/c1/sstate:webkitgtk:core2-64-poky-linux-musl:2.44.3:r0:core2-64:14:03c1e42313ff9e5a82a84a256b7ffbe7431e1dc0cc2d9253f48a65a2df2c7166_package_write_ipk.tar.zst
226M	./05/ce/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:05ce443d0a9418d1cb8b078bdf082df55ae7a700d6aae3f1c5444523ab27e25f_package_write_ipk.tar.zst
226M	./0a/b7/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:0ab78b85245212112f0bc4b90f4101ef2c750eb10c0b4aaad653c72337a33ed9_package_write_ipk.tar.zst
226M	./11/ed/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:11ed88ae2e48d498e5e117c0b46b375ffa07b3edb7f14b3bb91265f9702fc24e_package_write_ipk.tar.zst
226M	./13/c0/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:13c0e5db4f464e0fab8b98cca45b75f5a16e6052a743b46e57427469eece2681_package_write_ipk.tar.zst
226M	./1b/17/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:1b17f1d13298f652a272a62fab8e8bd983db16b1c2964958d94a112488649f1b_package_write_ipk.tar.zst
226M	./2b/21/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:2b21f7973cfa393478814d1e68a95a77b0ee383460b39679a79524003a66771c_package_write_ipk.tar.zst
226M	./2d/ec/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:2dec947e2b6b7fe87b67c82e75d39933480d03a52f4c593f59d78c7a4f6f722a_package_write_ipk.tar.zst
226M	./3a/b9/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:3ab9dd4fba267f48defbba007cc82090edca93fa82f6f36e0bbf6484b2a98d46_package_write_ipk.tar.zst
226M	./40/0f/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:400f84f8b3581f933679ec320c277f07c6906c3bbbe30d6f8cf117c542ac1a97_package_write_ipk.tar.zst
226M	./43/d9/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:43d94c539ab57cd8db8ceb7973188240d313d4f8417f70f274bc95998b5aff9c_package_write_ipk.tar.zst
226M	./46/cd/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:46cd07f5f5d3cad1957abbe28d560a066eea9f2456ccd0a0347b669911628877_package_write_ipk.tar.zst
226M	./58/b0/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:58b0dce17ca1f9be090a89213a9f0a482981de4ddc80fa7e4acc16abe7ef07be_package_write_ipk.tar.zst
226M	./71/20/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:71209c0fe79b678f080a95be6b7282a6805588a338ed29e8e64b28ec595b5065_package_write_ipk.tar.zst
226M	./74/3f/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:743f42bddc1342236c02c006ef505c01d89d67824af9eb0ff40c95a4fd6ffccc_package_write_ipk.tar.zst
226M	./81/d2/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:81d218ae602b359df3a73b7dce439391c847248eabb30f6da3c9606e8da3c4a6_package_write_ipk.tar.zst
226M	./95/db/sstate:webkitgtk:core2-64-poky-linux-musl:2.44.3:r0:core2-64:14:95dbd6ed8471595267cd38ef6a81d33a9ace92d7b106bb8c20aefd634c7fa894_package_write_ipk.tar.zst
226M	./9b/71/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:9b71805bfcaa22718c85adf2ddceef25e1d11a435e0859fe5ebff4596da063b2_package_write_ipk.tar.zst
226M	./9c/d3/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:9cd3aea0b9d575ca4f534d73cf092c9818addbd1e431fcadb245a75719698f97_package_write_ipk.tar.zst
226M	./a2/7c/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:a27c2cb870e369e702534fbd79b05b09fbfb0083a209deb200bea6bcdd445bfc_package_write_ipk.tar.zst
226M	./a7/6b/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:a76b97c7eb34a26f8af27331a928c0a31c9d9960063dba554ae4b5c83a648af1_package_write_ipk.tar.zst
226M	./b1/18/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:b11882b30acf04eea814160f81f1f31cc439957a1067672b60dc020454cab4f8_package_write_ipk.tar.zst
226M	./ba/22/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:ba22beec64ecf7c9abfef4bccc3acb5c30512c8c9bfdce542a8b804ebe5f0c1f_package_write_ipk.tar.zst
226M	./bd/dc/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:bddc2e42d196cccbdf766aa3837e197b0f1260282e454f1c85167a2a269ef615_package_write_ipk.tar.zst
226M	./c3/1e/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:c31e1236bbbc1ea57b4e842b335a96b5e2a43ff6d90fd1004c156032609461c8_package_write_ipk.tar.zst
226M	./c8/ab/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:c8ab24dc2617f1976cc1f90608ec1985cb6790ae84a1ba027d40432014063ead_package_write_ipk.tar.zst
226M	./cc/f8/sstate:webkitgtk:core2-64-poky-linux-musl:2.44.3:r0:core2-64:14:ccf8f81ace196e7064861a5f676f531e0acd9c3dd5036161204cabf447e046db_package_write_ipk.tar.zst
226M	./cd/34/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:cd3497a4840818e73a91e7c8b19f324e6ebe530f0241186578d77cdee0e0fbbc_package_write_ipk.tar.zst
226M	./d0/69/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:d069fb9e1fd3bef04bc5178d429c1c687b0137e16ee53599ab2faf22f95be7d4_package_write_ipk.tar.zst
226M	./d1/f3/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:d1f34a334eb15c88f33a7339e757ad868e501451d16011ab8f6bbe696a34198d_package_write_ipk.tar.zst
226M	./d3/62/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:d3621430453308b863dad1397ea1f2373f8d225882ad19145b22868d27a7c36e_package_write_ipk.tar.zst
226M	./d9/fd/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:d9fd11ca904131c1014f754f7baea44f79a197d93bf0435172de8f12718fa6f7_package_write_ipk.tar.zst
226M	./db/cb/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:dbcbc29856edcc13a52de52acc671534911c21e8797a86774d56e340a160daae_package_write_ipk.tar.zst
226M	./e4/72/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:e47226b587c85cc317c1709be6f6fba531ed7fe862d373204a0e21e01f50a380_package_write_ipk.tar.zst
226M	./e4/85/sstate:webkitgtk:core2-64-poky-linux-musl:2.44.3:r0:core2-64:14:e48529e863514d8a14aae07bf7dc7dcfe4d53fe9d0def0b018091d09df070e4e_package_write_ipk.tar.zst
226M	./e7/e6/sstate:webkitgtk:core2-64-poky-linux-musl:2.44.3:r0:core2-64:14:e7e6f49a9e1108b49381667f5362fe0ac7e175b83557b813dd23b9a74b0ffaaf_package_write_ipk.tar.zst
226M	./ea/f7/sstate:webkitgtk:core2-64-poky-linux-musl:2.44.3:r0:core2-64:14:eaf70d42ba38c1647f0ae99ee6534f10e557c6a15745db0a8398d9c114045571_package_write_ipk.tar.zst
226M	./f0/d5/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:f0d5061eb8dac07fafdbf2ef31be5e41e273af39a9d82f11af97a54998087439_package_write_ipk.tar.zst
226M	./fd/50/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:fd5030ccd1a1830b988c2c24bd1a725d546158fcb814a8b7eec6d831854beac5_package_write_ipk.tar.zst
227M	./16/7d/sstate:webkitgtk:core2-64-poky-linux-musl:2.44.3:r0:core2-64:14:167d29677f1b381b19a34b5dba3a5aa7e815a54961bffcde1bc617431c13f2ab_package_write_rpm.tar.zst
227M	./46/bf/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:46bfbb57075c01de8e8d488d0ff6f724be342aea217ad9bd939c16b18e5980b7_package_write_rpm.tar.zst
227M	./60/54/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:605457ac16f0ad029c5238c85896b7c62886142f17e7a557a77737cbef87025f_package_write_rpm.tar.zst
227M	./89/a5/sstate:webkitgtk:core2-64-poky-linux-musl:2.44.3:r0:core2-64:14:89a580f541accd6f8edb71cb554f415e7daa82dacae74e397a1bc7b25e17e4d3_package_write_rpm.tar.zst
227M	./a3/80/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:a38046fa04421541ea48aa69aa770bcf4846698360c296e1ea1feba008dc6ed5_package_write_rpm.tar.zst
227M	./c0/35/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:c035c0a7a941285fcbe1ee1f5264d766d30dceb4804773ac54f9e4d4e9a1440c_package_write_rpm.tar.zst
227M	./c0/8d/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:c08d87dc9a463627858bea2867b5606bba89b83403b2b64b79638c1c976fe571_package_write_rpm.tar.zst
227M	./d7/97/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:d797b52cc99bf5c83fd4ea1f29c01985967e3af1129ecb7fe5993c52599e00d7_package_write_rpm.tar.zst
227M	./f1/40/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:f1403cb75cfe0cc81eed276c889da02fcf9059b7db6b489b01e7b763f8aba674_package_write_rpm.tar.zst
227M	./f7/5a/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:f75a9bdd649cfcf2e57c770b0905c811881d8ef8cc45c96e069d93242946314f_package_write_rpm.tar.zst
228M	./2c/59/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:2c59c1d5ffef56415da6995ba5eb9018a48ac28876955548c2d23fdd4364c447_package_write_ipk.tar.zst
228M	./4e/58/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:4e583054602af21b15ea40b9f78aa1a54a9a5cdd5de1a5f9985b4ea25bda5df2_package_write_ipk.tar.zst
228M	./82/97/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:829796416b28407234773954ad46a3b7e47d611a8609c202c6f2d8f7181673a9_package_write_ipk.tar.zst
228M	./91/00/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:910049b38246ca82f49d26c6f0ae20fe7a446bb49b6a80bf08da7ffcacdd1661_package_write_ipk.tar.zst
228M	./9a/ab/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:9aabc1774a07b93d7dd437da822fa752fefe4207b58be28d3164f04930405588_package_write_ipk.tar.zst
228M	./df/86/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:df86e4e5d47edf3728e30aba6851a195e104c0eb1d456b91666cf4a557e1ad78_package_write_ipk.tar.zst
229M	./1a/ff/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:1aff95964474bb8a52b44059257aece09cb4dde2a0e2b0250e0b6ade9d5f960c_package_write_ipk.tar.zst
229M	./ac/ed/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:acedcdcb09253b11bfa22c9cc07273e8a131eea1ed2380f0157472dfd1735ac9_package_write_rpm.tar.zst
229M	./f8/17/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:14:f81787610d87e0ba0bc0cd378a9d25726d7c5b4c625e64394972a715b60c1bbe_package_write_rpm.tar.zst
829M	./01/24/sstate:webkitgtk:core2-32-poky-linux-musl:2.36.8:r0:core2-32:10:0124b257d01795b73f69cb171e512f947ce1b3d1973ac92f3957b91f0226205a_package_write_deb.tar.zst
829M	./0f/fa/sstate:webkitgtk:core2-32-poky-linux-musl:2.36.8:r0:core2-32:10:0ffaae0583756fb2a00e72bec7fd065f6aa08f03b5d7edf5127559d4610db5d3_package_write_ipk.tar.zst
829M	./6a/59/sstate:webkitgtk:core2-32-poky-linux-musl:2.36.8:r0:core2-32:10:6a5972cfda295dc362c03258e1d30cdc0b2cef02c119e26aae6fb2da714d22fd_package_write_ipk.tar.zst
829M	./aa/36/sstate:webkitgtk:core2-32-poky-linux-musl:2.36.8:r0:core2-32:10:aa36d2017242896ef5a27c83470dd6e645180ba37d35ee05e8d8a0c87d255bfe_package_write_ipk.tar.zst
829M	./c6/0e/sstate:webkitgtk:core2-32-poky-linux-musl:2.36.8:r0:core2-32:10:c60e8ede63e328099ca561a1f45571ec6239bcfb2b2d999f749ecd716940e1e2_package_write_deb.tar.zst
829M	./df/a7/sstate:webkitgtk:core2-32-poky-linux-musl:2.36.8:r0:core2-32:10:dfa74a1ab67dff9ec19f13ae14848b369b0b16ffda887d7e795bbf32ebb056ac_package_write_deb.tar.zst
925M	./4d/ee/sstate:webkitgtk:core2-32-poky-linux-musl:2.36.8:r0:core2-32:10:4dee19a640f9c9d2d8f99a23ea86630f2c4ae5f1ffdc774f7c2a4c6dee984a23_package_write_rpm.tar.zst
925M	./67/6a/sstate:webkitgtk:core2-32-poky-linux-musl:2.36.8:r0:core2-32:10:676a598f94d39938d9182847310909d610204ab144e397fdb2b4aed754f78bdf_package_write_rpm.tar.zst
925M	./d0/6a/sstate:webkitgtk:core2-32-poky-linux-musl:2.36.8:r0:core2-32:10:d06ae3ace9b0871a3dd8697fd7711a9dd55032b9a1e583bf63c6a90e7f9b589e_package_write_rpm.tar.zst
991M	./26/92/sstate:webkitgtk:core2-64-poky-linux:2.36.8:r0:core2-64:10:269254daa4a28faf26f7bf78024c53c271967c29f5def824c428c71eb444a1e6_package_write_ipk.tar.zst
991M	./2a/67/sstate:webkitgtk:core2-64-poky-linux:2.36.8:r0:core2-64:10:2a67734eab548d1ad045bb7dc34b9c61931f8be011a39074dcaecff5d0ca1c96_package_write_deb.tar.zst
991M	./44/9f/sstate:webkitgtk:core2-64-poky-linux-musl:2.36.8:r0:core2-64:10:449f8d1e01af96901d1693589c8d77e63efebb422fa28247c6f384e8cefbce17_package_write_deb.tar.zst
991M	./60/ae/sstate:webkitgtk:core2-64-poky-linux:2.36.8:r0:core2-64:10:60aea1cbf52dc14a6bf0dbace7b44567a717db9cedce3dae5b65575e838a6c5b_package_write_ipk.tar.zst
991M	./61/af/sstate:webkitgtk:core2-64-poky-linux:2.36.8:r0:core2-64:10:61af36071b8c3b8dba7cfaf04d40aa763a75ddc45d2605f7ef3bac3198b4cf40_package_write_deb.tar.zst
991M	./80/7e/sstate:webkitgtk:core2-64-poky-linux:2.36.8:r0:core2-64:10:807e146ff767a8b6c0afd47c33b37d3230f012cbb2c74dd2a0290f2e0cea38d0_package_write_ipk.tar.zst
991M	./81/13/sstate:webkitgtk:core2-64-poky-linux-musl:2.36.8:r0:core2-64:10:81131c2a64d3766b142989feac7dcc23b596adc7db9cbfc6ef48aeeeb077a896_package_write_ipk.tar.zst
991M	./9a/31/sstate:webkitgtk:core2-64-poky-linux:2.36.8:r0:core2-64:10:9a31865b58eff1a722a70f7e10a10cc5dc16190ba0e9e9f25de7b76add8fda6f_package_write_deb.tar.zst
991M	./b0/39/sstate:webkitgtk:core2-64-poky-linux-musl:2.36.8:r0:core2-64:10:b039877427b38d15f258080ae29c957cd3f0b595d2e2c84d55f26cee5aa16f11_package_write_deb.tar.zst
991M	./b5/64/sstate:webkitgtk:core2-64-poky-linux:2.36.8:r0:core2-64:10:b56443d9bd458a8b941ccf10b53d8e2dc6b1f2076dee1f98afd31aba38b0d7ca_package_write_deb.tar.zst
991M	./ca/26/sstate:webkitgtk:core2-64-poky-linux-musl:2.36.8:r0:core2-64:10:ca26de97cc0b98c445215f2c1051e14aa8919a880761bd3a88ab5b3cc4c0e358_package_write_ipk.tar.zst
991M	./e3/ef/sstate:webkitgtk:core2-64-poky-linux:2.36.8:r0:core2-64:10:e3ef8e1df715ceb62426337d48c9f2e7c63085eecb08bfe04ac2d0d875cb4352_package_write_ipk.tar.zst
992M	./31/d4/sstate:webkitgtk:core2-64-poky-linux:2.36.8:r0:core2-64:10:31d4f644a2cb34a2d0658cc7a13467feb88f9960180cad8b9faf10bceec1a31d_package_write_deb.tar.zst
992M	./39/5b/sstate:webkitgtk:core2-64-poky-linux:2.36.8:r0:core2-64:10:395bd88e1a2dd6028724fc6c8039b2e432c3a1eae8ee71fd562f86acd7ccd41f_package_write_ipk.tar.zst
992M	./46/56/sstate:webkitgtk:core2-64-poky-linux:2.36.8:r0:core2-64:10:4656f449787cad1d29dabb020d53aedada8948605c607312c8c18afc108a1c76_package_write_ipk.tar.zst
992M	./82/71/sstate:webkitgtk:core2-64-poky-linux:2.36.8:r0:core2-64:10:8271dfad2362e8fdc6fa14642a5abf37f061df589ab62721ed2b5deb1674805d_package_write_deb.tar.zst
992M	./d6/53/sstate:webkitgtk:core2-64-poky-linux:2.36.8:r0:core2-64:10:d653fff3801f698b4e65c9ef22f6c6b5ddf924c7782f125025ce02579e37798f_package_write_deb.tar.zst
992M	./e4/0e/sstate:webkitgtk:core2-64-poky-linux:2.36.8:r0:core2-64:10:e40e2e20222fcc48698d1b22c6198dd18893f74ca0bfe270f51ca88380a8ad0d_package_write_deb.tar.zst
992M	./fc/d1/sstate:webkitgtk:core2-64-poky-linux:2.36.8:r0:core2-64:10:fcd1d807d27c3d74c95f9968c9ecdb689c4296cbb0bb578e7a834614a06556e1_package_write_deb.tar.zst
993M	./1e/62/sstate:webkitgtk:core2-64-poky-linux:2.36.8:r0:core2-64:10:1e62b5a1c8289993d45ad48793f0a7329940116ba7d23e574fe642133cff90b2_package_write_ipk.tar.zst
993M	./d8/4d/sstate:webkitgtk:core2-64-poky-linux:2.36.8:r0:core2-64:10:d84d561e2960cb1545d611bffbdfecb3098a7bfe49c000c27ab0567d78f882b4_package_write_ipk.tar.zst
993M	./ec/5c/sstate:webkitgtk:core2-64-poky-linux:2.36.8:r0:core2-64:10:ec5c6ac2746dd7b9cd8a91ebdff1cbd81e730f340d7eff40c1a2ad1cf23cfcac_package_write_ipk.tar.zst
1.1G	./08/ca/sstate:webkitgtk:core2-32-poky-linux-musl:2.44.3:r0:core2-32:12:08caa33d57a8f5a745d6c28e1656279a3cfe224592b1f507193ca80fef43b011_package_write_deb.tar.zst
1.1G	./14/1d/sstate:webkitgtk:core2-64-poky-linux:2.36.8:r0:core2-64:10:141da7a488410ac67bae9aaf3d71d4abef18434405f4e5275cda92d4e071be95_package_write_rpm.tar.zst
1.1G	./16/52/sstate:webkitgtk:core2-64-poky-linux:2.36.8:r0:core2-64:10:16529dc6e8723a1107ddcda80af2f9d7a15e54f06780b0f94cc00aa84e7bdfc9_package_write_rpm.tar.zst
1.1G	./43/c2/sstate:webkitgtk:core2-64-poky-linux:2.36.8:r0:core2-64:10:43c28848bd390deb844628830b208cdca6c23ed49b7b03e75457ed48c3d7a156_package_write_rpm.tar.zst
1.1G	./60/f0/sstate:webkitgtk:core2-64-poky-linux:2.36.8:r0:core2-64:10:60f087b8029c920b2a89cb4628bf730e4274e904ca3917ad6bbd0a13d998d93e_package_write_rpm.tar.zst
1.1G	./72/2d/sstate:webkitgtk:core2-64-poky-linux-musl:2.36.8:r0:core2-64:10:722df891d1779508d44f4a05966131bbd878e8c0700d9679a03e5791f8ba5966_package_write_rpm.tar.zst
1.1G	./72/c3/sstate:webkitgtk:core2-64-poky-linux:2.36.8:r0:core2-64:10:72c3fc5f2eda8c6958789977e7a849bee6efe6281d70c5abc20f9632eed9ccfa_package_write_rpm.tar.zst
1.1G	./bc/a4/sstate:webkitgtk:core2-64-poky-linux:2.36.8:r0:core2-64:10:bca45e3769b2d4e10a18c685e7eb0d949413b8136ca7632ce62f4b64d545391c_package_write_rpm.tar.zst
1.1G	./e6/a1/sstate:webkitgtk:core2-64-poky-linux:2.36.8:r0:core2-64:10:e6a121f7b5d3931b8c49da8087741076f987299844c61fd61bef45f66f71d1c0_package_write_rpm.tar.zst
1.1G	./ee/e5/sstate:webkitgtk:core2-64-poky-linux-musl:2.36.8:r0:core2-64:10:eee5c04bd1bf087af25c2dd7a59552179dfbe32f319e7459bf38bbb15ac65dc5_package_write_rpm.tar.zst
1.1G	./ef/5b/sstate:webkitgtk:core2-32-poky-linux-musl:2.44.3:r0:core2-32:12:ef5b23479702c215939163db7657e88b7dd33e0609c2a222fd3bb5753c7c3079_package_write_deb.tar.zst
1.1G	./ef/c2/sstate:webkitgtk:core2-64-poky-linux:2.36.8:r0:core2-64:10:efc28fd28e0e72ef7ca3ee441f13dabe54acf9abf9625f5ee8dff23352c2118f_package_write_rpm.tar.zst
1.1G	./f7/4d/sstate:webkitgtk:core2-64-poky-linux:2.36.8:r0:core2-64:10:f74d53b90bd331a44066b58a6962d522938d60082e39e9484a7d66ee9befa68f_package_write_rpm.tar.zst
1.2G	./19/96/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:12:1996743ba99502d51b3ee1d4948b9bac752f3c3fcef996a09478ceff15103922_package_write_deb.tar.zst
1.2G	./52/0d/sstate:webkitgtk:core2-64-poky-linux-musl:2.44.3:r0:core2-64:12:520d87913b6e6de4c44a95172677a2fdcaed9e7b0e8fd092feeb1983b1499961_package_write_deb.tar.zst
1.2G	./54/aa/sstate:webkitgtk:core2-32-poky-linux-musl:2.44.3:r0:core2-32:12:54aac35e9dbf3079ab7177df4ede2d04d2ad14f795f1d4a7cf0c8381d02d4f8d_package_write_rpm.tar.zst
1.2G	./9a/ad/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:12:9aadbf2fd3655a8da5f59f1d5acd085a47379cf591c35f6a00b3f47c04aef40a_package_write_deb.tar.zst
1.2G	./bf/c5/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:12:bfc55c20b7b3d145ad135f60c47980432988595f6f5d82831331056e6d8d38a7_package_write_deb.tar.zst
1.2G	./d2/e3/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:12:d2e319117f1dfe0ea571f0768f6ca112dc3d182cf4fe65e1b61c4b01a7e0c716_package_write_deb.tar.zst
1.2G	./d9/1e/sstate:webkitgtk:core2-32-poky-linux-musl:2.44.3:r0:core2-32:12:d91e7efe78cb28c7852ca2ba4cedf9ceb48b22c7fd3928e5b99c9a090764a8eb_package_write_rpm.tar.zst
1.2G	./fa/54/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:12:fa5410f67280df75174e3bc41165bbfb29cd971d55a0880ff0f25912648029b5_package_write_deb.tar.zst
1.4G	./1b/c4/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:12:1bc4254e69d6f85ee0995220875d89081ac886ca97ff6d99a00e4d061b32e013_package_write_rpm.tar.zst
1.4G	./28/3d/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:12:283dd99a91fa4285be75ec5a72271f9b96e3d5a0b9c9792f63630efbf985fecc_package_write_rpm.tar.zst
1.4G	./4b/5a/sstate:webkitgtk:core2-64-poky-linux-musl:2.44.3:r0:core2-64:12:4b5af4b6f991cda2b3ba217fc18f6a80876b2ba2742bf4db6c1585f98a317403_package_write_rpm.tar.zst
1.4G	./4d/63/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:12:4d63e0526d82edfe3ed28f541b4f9e7ed3eadfeca0a51a66814f60f0cc2aecc4_package_write_rpm.tar.zst
1.4G	./5a/0f/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:12:5a0f8e37cd09d417620de3f5a6a63bc31a3b1654c59369b04dc51c75598ea99c_package_write_rpm.tar.zst
1.4G	./7b/3c/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:12:7b3c29777a4b514bfe1a51ff096c40916254e4535da9a04eb8ca7e163caa1e97_package_write_rpm.tar.zst
1.6G	./6f/90/sstate:webkitgtk:core2-32-poky-linux-musl:2.44.3:r0:core2-32:12:6f90e03f5d508dd9576c5c33dbc9e85cd8578e1fe53b5aa85356cfcf41071121_package_write_ipk.tar.zst
1.6G	./f4/8b/sstate:webkitgtk:core2-32-poky-linux-musl:2.44.3:r0:core2-32:12:f48b3f5016b861a67d5c49a67f422ef50afee48d10984e586bb9e422af77096a_package_write_ipk.tar.zst
1.9G	./0b/4e/sstate:webkitgtk:core2-64-poky-linux:2.36.8:r0:core2-64:10:0b4e5fbb24977a22dd87d5a07f4479fdc881877f471e61838b2c78e69f566e4f_package_write_ipk.tar.zst
1.9G	./18/13/sstate:webkitgtk:core2-64-poky-linux:2.36.8:r0:core2-64:10:18133e48b0287e34b9ed621bb7b8be900cadf28570626544fe088e1a9b0ba8c6_package_write_deb.tar.zst
1.9G	./57/41/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:12:5741b3f60783bc00ac8d262dd3f0cc7fc8610bc68b5754f25e599404b4bf74f4_package_write_ipk.tar.zst
1.9G	./61/c2/sstate:webkitgtk:core2-64-poky-linux:2.36.8:r0:core2-64:10:61c2b8176223deb9057040867d397005ff69947a63ede10c4a23e0d712597449_package_write_ipk.tar.zst
1.9G	./6e/da/sstate:webkitgtk:core2-64-poky-linux:2.36.8:r0:core2-64:10:6eda185d1b18aa79a90fb63f9ce1627615e714858d9e78a258e82a9b24870d8a_package_write_deb.tar.zst
1.9G	./85/d2/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:12:85d2bdf67506d6aee96cfc152f4dca8fa2078e43f7a882f4182e68748c07883c_package_write_ipk.tar.zst
1.9G	./c8/66/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:12:c86613497c75b0e301614ad48a6f2071ff887f4cba7e28c292bea78fa3827d63_package_write_ipk.tar.zst
1.9G	./d9/75/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:12:d9753bf17cfd97d8f8dbe8f5ff1e0c9c16c4a95337a023c67fadf4f15c2986e6_package_write_ipk.tar.zst
1.9G	./da/f3/sstate:webkitgtk:core2-64-poky-linux:2.44.3:r0:core2-64:12:daf38416352486a85d2e5d36795a01e9f1b7f372e78676aaa62a81db84a769ae_package_write_ipk.tar.zst
1.9G	./db/c2/sstate:webkitgtk:core2-64-poky-linux-musl:2.44.3:r0:core2-64:12:dbc25bfbf4deb428afd0c8dea5c1480cd6c4594b80225e8075ae881cb1e3f534_package_write_ipk.tar.zst
2.1G	./66/8a/sstate:webkitgtk:core2-64-poky-linux:2.36.8:r0:core2-64:10:668ac9f7df44bf1d19060e92100e751f6c64f79827a3dae0f1eed06b0c9eabf6_package_write_rpm.tar.zst
2.1G	./a7/bd/sstate:webkitgtk:core2-64-poky-linux:2.36.8:r0:core2-64:10:a7bdd1d1e2536dd9d34c57e8d36e80096e519a1472a0fa621eca90f05a29b733_package_write_rpm.tar.zst

Re: [OE-core] Space optimizations in kirkstone and scarthgap for sstate objects

[Mikko Rapeli]

Hi,

On Thu, Nov 21, 2024 at 10:41:29AM +0000, Richard Purdie via lists.openembedded.org wrote:
> We've been trying to resolve some of our sstate CDN issues and one
> thing causing us big headaches are large sstate files. By far the worst
> offenders are webkitgtk and llvm.
> 
> In master (and in styhead) we have these:
> 
> https://git.yoctoproject.org/poky/commit/?id=d9066258a1e48e75cf814fb0940ac420bcdd6e99
> https://git.yoctoproject.org/poky/commit/?id=aca8acceb87e568115da9696732e915d91da9652
> https://git.yoctoproject.org/poky/commit/?id=1d8d9b1f87b21af36d9c328132f71c528eefd5b0
> 
> which did significantly reduce the size of the output. Unfortunately I
> made the first two work with changes to DEBUG_LEVELFLAG which are not
> suited for backporting:
> 
> https://git.yoctoproject.org/poky/commit/?id=1cf0974ad242f7eb2815a4ef0e3e5b6507ca56ea
> https://git.yoctoproject.org/poky/commit/?id=ae4ec59b3eff85aaa1cfe914a134e56e1d125744
> 
> I do think we should apply a similar tweak to kirkstone and scarthgap
> though. The third patch above shouldbackport. We should be able to do
> something equivalent for llvm. Any thoughts?
> 
> Michael obtained some sstate size info which I've shared below. You can
> tell the age by the sstate version number (10, 12 and 14 are kirkstone,
> scarthgap and styhead/master) and the recipe versions.

Can similar changes be done via the CI build scripts in the affected branches,
without modifying poky?

I would not mind breaking stable rules when it's about keeping upstream
infrastructure sane and running. Details can be documented to users if they
are impacted and need to revert those changes in their configuration.

Would be nice to know what kind of issues these are in detail and what is the CDN
implementation.

Cheers,

-Mikko

Re: [OE-core] Space optimizations in kirkstone and scarthgap for sstate objects

[Richard Purdie]

On Thu, 2024-11-21 at 13:00 +0200, Mikko Rapeli wrote:
> Hi,
> 
> On Thu, Nov 21, 2024 at 10:41:29AM +0000, Richard Purdie via lists.openembedded.org wrote:
> > We've been trying to resolve some of our sstate CDN issues and one
> > thing causing us big headaches are large sstate files. By far the worst
> > offenders are webkitgtk and llvm.
> > 
> > In master (and in styhead) we have these:
> > 
> > https://git.yoctoproject.org/poky/commit/?id=d9066258a1e48e75cf814fb0940ac420bcdd6e99
> > https://git.yoctoproject.org/poky/commit/?id=aca8acceb87e568115da9696732e915d91da9652
> > https://git.yoctoproject.org/poky/commit/?id=1d8d9b1f87b21af36d9c328132f71c528eefd5b0
> > 
> > which did significantly reduce the size of the output. Unfortunately I
> > made the first two work with changes to DEBUG_LEVELFLAG which are not
> > suited for backporting:
> > 
> > https://git.yoctoproject.org/poky/commit/?id=1cf0974ad242f7eb2815a4ef0e3e5b6507ca56ea
> > https://git.yoctoproject.org/poky/commit/?id=ae4ec59b3eff85aaa1cfe914a134e56e1d125744
> > 
> > I do think we should apply a similar tweak to kirkstone and scarthgap
> > though. The third patch above shouldbackport. We should be able to do
> > something equivalent for llvm. Any thoughts?
> > 
> > Michael obtained some sstate size info which I've shared below. You can
> > tell the age by the sstate version number (10, 12 and 14 are kirkstone,
> > scarthgap and styhead/master) and the recipe versions.
> 
> Can similar changes be done via the CI build scripts in the affected branches,
> without modifying poky?

In theory, yes. I'm not sure we want to "hide" this kind of change
though.

> I would not mind breaking stable rules when it's about keeping upstream
> infrastructure sane and running. Details can be documented to users if they
> are impacted and need to revert those changes in their configuration.

If we do make a change, users would easily be able to reverse it
locally if needed.

> Would be nice to know what kind of issues these are in detail and what is the CDN
> implementation.

Most CDNs have object size limits so the setup cascades through
different levels of completely different systems to cope with our large
object sizes. The CDNs themselves are not designed for large objects
either, in their view we should split these into multiple smaller
objects. That isn't really possible without rewriting the way bitbake
works with sstate.

Objects are only pulled into the CDN at first request too, the larger
the objects, the longer this takes. The setup has changed a bit as
we've tried to work out how we can make it work. One issue was the size
of the "pipe" between our servers and the CDN nodes. The new
infrastructure/data centre location has resolved that potential
bottleneck.

Cheers,

Richard

Re: [OE-core] Space optimizations in kirkstone and scarthgap for sstate objects

[Mikko Rapeli]

Hi,

On Thu, Nov 21, 2024 at 11:10:11AM +0000, Richard Purdie wrote:
> On Thu, 2024-11-21 at 13:00 +0200, Mikko Rapeli wrote:
> > Hi,
> > 
> > On Thu, Nov 21, 2024 at 10:41:29AM +0000, Richard Purdie via lists.openembedded.org wrote:
> > > We've been trying to resolve some of our sstate CDN issues and one
> > > thing causing us big headaches are large sstate files. By far the worst
> > > offenders are webkitgtk and llvm.
> > > 
> > > In master (and in styhead) we have these:
> > > 
> > > https://git.yoctoproject.org/poky/commit/?id=d9066258a1e48e75cf814fb0940ac420bcdd6e99
> > > https://git.yoctoproject.org/poky/commit/?id=aca8acceb87e568115da9696732e915d91da9652
> > > https://git.yoctoproject.org/poky/commit/?id=1d8d9b1f87b21af36d9c328132f71c528eefd5b0
> > > 
> > > which did significantly reduce the size of the output. Unfortunately I
> > > made the first two work with changes to DEBUG_LEVELFLAG which are not
> > > suited for backporting:
> > > 
> > > https://git.yoctoproject.org/poky/commit/?id=1cf0974ad242f7eb2815a4ef0e3e5b6507ca56ea
> > > https://git.yoctoproject.org/poky/commit/?id=ae4ec59b3eff85aaa1cfe914a134e56e1d125744
> > > 
> > > I do think we should apply a similar tweak to kirkstone and scarthgap
> > > though. The third patch above shouldbackport. We should be able to do
> > > something equivalent for llvm. Any thoughts?
> > > 
> > > Michael obtained some sstate size info which I've shared below. You can
> > > tell the age by the sstate version number (10, 12 and 14 are kirkstone,
> > > scarthgap and styhead/master) and the recipe versions.
> > 
> > Can similar changes be done via the CI build scripts in the affected branches,
> > without modifying poky?
> 
> In theory, yes. I'm not sure we want to "hide" this kind of change
> though.

I understand, just wanted to show this as an option. Changes which are specific
to your infrastructure and which break stable rules could be considered
as specific to the CI build jobs you run, hence not hiding but keeping
such changes in CI scripts, which generate the build configurations
and matrixes and the local.conf's.

> > I would not mind breaking stable rules when it's about keeping upstream
> > infrastructure sane and running. Details can be documented to users if they
> > are impacted and need to revert those changes in their configuration.
> 
> If we do make a change, users would easily be able to reverse it
> locally if needed.
> 
> > Would be nice to know what kind of issues these are in detail and what is the CDN
> > implementation.
> 
> Most CDNs have object size limits so the setup cascades through
> different levels of completely different systems to cope with our large
> object sizes.�The CDNs themselves are not designed for large objects
> either, in their view we should split these into multiple smaller
> objects. That isn't really possible without rewriting the way bitbake
> works with sstate.
> 
> Objects are only pulled into the CDN at first request too, the larger
> the objects, the longer this takes. The setup has changed a bit as
> we've tried to work out how we can make it work. One issue was the size
> of the "pipe" between our servers and the CDN nodes. The new
> infrastructure/data centre location has resolved that potential
> bottleneck.

I've used "rsync sstate mirror from main server before build" as CDN
which did not break but underlying VMware virtual storage did break
under heavy bitbake load.

Cheers,

-Mikko