Re: [RFC] Per-file compression

[Tom Marshall <tom@cyngn.com>]

I'll try my hand at ecompressfs shortly, thanks!On Apr 20, 2015 9:51 AM, Richard Weinberger <richard@nod.at> wrote:
>
> Am 20.04.2015 um 16:53 schrieb Tyler Hicks: 
> > On 2015-04-18 17:07:16, Richard Weinberger wrote: 
> >> Hi! 
> >> 
> >> Am 18.04.2015 um 16:58 schrieb Tom Marshall: 
> >>> On Sat, Apr 18, 2015 at 01:41:09PM +0200, Richard Weinberger wrote: 
> >>>> On Sat, Apr 18, 2015 at 12:20 AM, Tom Marshall <tom@cyngn.com> wrote: 
> >>>>> So, I wrote a thing called 'zfile' that hooks into the VFS layer and 
> >>>>> intercepts file_operations to do file (de)compression on the fly. When a 
> >>>>> file is opened, it reads and decompresses the data into memory.  The file 
> >>>>> may be read, written, and mmaped in the usual way.  If the contents are 
> >>>>> changed, the data is compressed and written back.  A working patch for an 
> >>>>> older kernel version may be found at: 
> >>>>> http://review.cyanogenmod.org/#/c/95220/ 
> >>>> 
> >>>> So, I've extracted the patch from that website and gave a quick review. 
> >>>> 
> >>>> I'm pretty sure VFS folks will hate the VFS layering you do. 
> >>> 
> >>> This, I'm afraid, is the biggest obstacle to such a solution.  I know that 
> >>> OverlayFS has been merged, so filesystem stacking is acceptable.  Perhaps 
> >>> there would be a way to design a filesystem that stacks compression? 
> >> 
> >> That's why I said think of adding compression support to ecryptfs. 
> > 
> > I think adding compression support to eCryptfs is the wrong approach. 
> > The "X is already a stacked filesystem so look into adding compression 
> > support to it" logic also works when X=overlayfs. I doubt that Miklos 
> > would be willing to accept such a feature. :) 
>
> My thought was that compression is not far away from crypto an hence 
> a lot of ecryptfs could be reused. 
>
> > A stacked filesystem that implements compression should be fairly 
> > simple. If it is not simple, it is too complicated to try to wedge into 
> > an unrelated stacked filesystem. 
> > 
> > While it may be the quickest route to your end goal, it will overly 
> > complicate eCryptfs. eCryptfs already has plenty of complexity around 
> > the file offset since metadata may be stored in the first 8192 bytes of 
> > the lower file, which offsets the entire file, and the end of the file 
> > has to be padded. Mixing in compression would make things much worse. 
>
> I assumed that you need also some meta data for compression. 
> At least if you do it in a non-trivial way. 
>
> > Also, eCryptfs has lots of cruft that is definitely not needed for 
> > compression. 
>
> As you're the maintainer of ecryptfs you know obviously better than I do. :) 
> Tom, to make the story short, you'll have to experiment a bit. 
>
> Thanks, 
> //richard