Re: [RFC] Per-file compression

[Richard Weinberger <richard@nod.at>]

Am 20.04.2015 um 16:53 schrieb Tyler Hicks:
> On 2015-04-18 17:07:16, Richard Weinberger wrote:
>> Hi!
>>
>> Am 18.04.2015 um 16:58 schrieb Tom Marshall:
>>> On Sat, Apr 18, 2015 at 01:41:09PM +0200, Richard Weinberger wrote:
>>>> On Sat, Apr 18, 2015 at 12:20 AM, Tom Marshall <tom@cyngn.com> wrote:
>>>>> So, I wrote a thing called 'zfile' that hooks into the VFS layer and
>>>>> intercepts file_operations to do file (de)compression on the fly. When a
>>>>> file is opened, it reads and decompresses the data into memory.  The file
>>>>> may be read, written, and mmaped in the usual way.  If the contents are
>>>>> changed, the data is compressed and written back.  A working patch for an
>>>>> older kernel version may be found at:
>>>>> http://review.cyanogenmod.org/#/c/95220/
>>>>
>>>> So, I've extracted the patch from that website and gave a quick review.
>>>>
>>>> I'm pretty sure VFS folks will hate the VFS layering you do.
>>>
>>> This, I'm afraid, is the biggest obstacle to such a solution.  I know that
>>> OverlayFS has been merged, so filesystem stacking is acceptable.  Perhaps
>>> there would be a way to design a filesystem that stacks compression?
>>
>> That's why I said think of adding compression support to ecryptfs.
> 
> I think adding compression support to eCryptfs is the wrong approach.
> The "X is already a stacked filesystem so look into adding compression
> support to it" logic also works when X=overlayfs. I doubt that Miklos
> would be willing to accept such a feature. :)

My thought was that compression is not far away from crypto an hence
a lot of ecryptfs could be reused.

> A stacked filesystem that implements compression should be fairly
> simple. If it is not simple, it is too complicated to try to wedge into
> an unrelated stacked filesystem.
> 
> While it may be the quickest route to your end goal, it will overly
> complicate eCryptfs. eCryptfs already has plenty of complexity around
> the file offset since metadata may be stored in the first 8192 bytes of
> the lower file, which offsets the entire file, and the end of the file
> has to be padded. Mixing in compression would make things much worse.

I assumed that you need also some meta data for compression.
At least if you do it in a non-trivial way.

> Also, eCryptfs has lots of cruft that is definitely not needed for
> compression.

As you're the maintainer of ecryptfs you know obviously better than I do. :)
Tom, to make the story short, you'll have to experiment a bit.

Thanks,
//richard