Re: [RFC] Per-file compression

[Richard Weinberger <richard@nod.at>]

Hi!

Am 18.04.2015 um 16:58 schrieb Tom Marshall:
> On Sat, Apr 18, 2015 at 01:41:09PM +0200, Richard Weinberger wrote:
>> On Sat, Apr 18, 2015 at 12:20 AM, Tom Marshall <tom@cyngn.com> wrote:
>>> So, I wrote a thing called 'zfile' that hooks into the VFS layer and
>>> intercepts file_operations to do file (de)compression on the fly. When a
>>> file is opened, it reads and decompresses the data into memory.  The file
>>> may be read, written, and mmaped in the usual way.  If the contents are
>>> changed, the data is compressed and written back.  A working patch for an
>>> older kernel version may be found at:
>>> http://review.cyanogenmod.org/#/c/95220/
>>
>> So, I've extracted the patch from that website and gave a quick review.
>>
>> I'm pretty sure VFS folks will hate the VFS layering you do.
> 
> This, I'm afraid, is the biggest obstacle to such a solution.  I know that
> OverlayFS has been merged, so filesystem stacking is acceptable.  Perhaps
> there would be a way to design a filesystem that stacks compression?

That's why I said think of adding compression support to ecryptfs.

>> Beside of that you decompress the *whole* file into memory at open() time.
>> This will explode as soon you deal with bigger files.
> 
> I was thinking that a header with compressed offsets might be an option.  Or
> in the case of lz4 it's not terribly inefficient to scan the blocks.
> 
>> Also you seem to trust the user.compression.realsize xattr provided by
>> userspace.  That looks exploitable.
> 
> This is only used to provide a fast stat().  It could be put into a header
> or even removed entirely in favor of scanning the blocks.
> 
>> Back to my original question, why not FUSE?
> 
> Mostly because I'm not very familiar with FUSE.  But I suppose it could be
> an option.  I have some immediate concerns though:
> 
> * How would it affect performance?  FUSE passes all operations through user
>   space, correct?

Yeah, but you'll have to do benchmarks to find out the real trade off.
I'd give it a tray. Your targets are smartphones not HPC clusters.

> * How big might a reasonably complete implementation be for ARM?  The
>   implementation would need to be stored in the initrd.

I bet you can do it in less than 1000 lines of C...

>> Or add compression support to ecryptfs...
> 
> Several filesystems have native compression support.  But this would violate
> the goal of not switching filesystems.

...your goals. Be flexible. ;)
BTW: ecryptfs is an overlay filesystem.

Thanks,
//richard