Re: sstate compression

[Richard Purdie <richard.purdie@linuxfoundation.org>]

On Wed, 2012-01-04 at 15:05 +0000, Phil Blundell wrote:
> Has anybody experimented with the effects of using different/no
> compression for the sstate packages? 
> 
> I happened to notice that, for one of my builds, oe was spending a
> noticeable amount of time gzipping the tarfiles for sstate.  Obviously
> this indicates that I should get myself a better computer, but I do
> wonder whether there is a different tradeoff that can be made here.
> 
> Taking one of my webkit sstate archives as an example, the uncompressed
> tarfile is 112M. 
> 
> gzip (default compression) reduces this to 28M in about 4.2 seconds.
> gzip -1 reduces it to 33M in about 1.75 seconds.
> lzop reduces it to 40M in about 0.75 seconds.
> 
> Presumably the same sort of considerations apply to the compressed data
> inside .ipks, though of course the file sizes there tend to be rather
> smaller so I guess the impact is less.
> 
> I'm currently setting GZIP="-1" in my local configuration since that's
> easy to arrange and seems to produce a worthwhile benefit.  It's not
> totally obvious to me whether switching to lzo would be a net win or a
> loss, but either way this would involve hacking sstate.bbclass and I was
> too lazy to do that so far.

There has already been a patch allowing different compression mechanisms
from Matthew posted on this mailing list. Its not been merged yet as I'd
really prefer one format and we didn't (and still don't) have good
statistics about what the tradeoffs are.

Where the sstate tarballs are hitting the network, size probably is more
important than speed. There are also a number of oustanding bugs todo
with bootstrap of sstate alongside gzip-native, tar-native and so on
since things can fail when these are half installed in a sysroot and
something executes using them :/.

Cheers,

Richard