Re: [RFC] Per-file compression

[Alex Elsayed <eternaleye@gmail.com>]

Theodore Ts'o wrote:

> On Sat, Apr 18, 2015 at 10:06:14AM +0200, Richard Weinberger wrote:
>> On Sat, Apr 18, 2015 at 12:20 AM, Tom Marshall <tom@cyngn.com> wrote:
>> > We are running into space issues when upgrading Android devices to the
>> > latest version.  This issue is particularly troublesome for 64-bit
>> > devices that initially shipped without 64-bit support, as the multi-lib
>> > files nearly
>> > double the space requirements.  As it happens, many system files are
>> > quite
>> > compressible -- namely shared libs and apk files.  So implementing
>> > filesystem compression is a natural solution.
> 
> Tom, have you verified whether or not this approach actually is
> sufficient to address the problem you are trying to solve?
> Specifically, (a) can you get enough space compression that it will
> make things fit?  Given that apk files are already compressed using
> zip, I'm a bit surprised you would get enough of a compression savings
> that it's going to allow to make things fit.  And secondly (b) after
> you decompress the shared libraries, is there enough memory that the
> resulting system functions well.  Often times these older Android
> devices don't have a huge amount of memory, and so either with or
> without the overhead of needing to decompress the entire file (which
> can be solved with chunking, although this tends to reduce the
> compression efficiency), is the system going to function well using
> 64-bit binaries that take up a lot more room than the older 32-bit
> binaries?

APK files may be using the zip _format_, but it's not uncommon for the 
compression level to be zero. Honestly, _that_ sounds like a saner way of 
addressing the issue to me, especially if files are compressed whole rather 
than chunked for random access - set the compression level to 9 or 
something, since whichever level you add the compression at you've already 
lost the (IMO of dubious importance) speed benefit zero-compression zip 
gives.

>> > So, I wrote a thing called 'zfile' that hooks into the VFS layer and
>> > intercepts file_operations to do file (de)compression on the fly. When
>> > a
>> > file is opened, it reads and decompresses the data into memory.  The
>> > file
>> > may be read, written, and mmaped in the usual way.  If the contents are
>> > changed, the data is compressed and written back.
> 
> Do you really need to be able to rewrite the files?  Life gets much
> easier if you can assume that the compressed files are read-only;
> especially once you start trying to use chunking, it *definitely*
> becomes easier if you don't need to support modifying the compressed
> files.  And in the case of the system partition, this might be a safe
> assumption,yes?
> 
> - Ted
> --
> To unsubscribe from this list: send the line "unsubscribe linux-fsdevel"
> in the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html