From mboxrd@z Thu Jan 1 00:00:00 1970 From: Tyler Hicks Subject: Re: [RFC] Per-file compression Date: Mon, 20 Apr 2015 09:53:57 -0500 Message-ID: <20150420145357.GA10981@boyd> References: <55318733.4000503@cyngn.com> <20150418145828.GA13809@eden.sea.cyngn.com> <55327324.3030400@nod.at> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="y0ulUmNC+osPPQO6" Cc: Tom Marshall , linux-fsdevel To: Richard Weinberger Return-path: Received: from youngberry.canonical.com ([91.189.89.112]:50348 "EHLO youngberry.canonical.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1755083AbbDTOyD (ORCPT ); Mon, 20 Apr 2015 10:54:03 -0400 Content-Disposition: inline In-Reply-To: <55327324.3030400@nod.at> Sender: linux-fsdevel-owner@vger.kernel.org List-ID: --y0ulUmNC+osPPQO6 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On 2015-04-18 17:07:16, Richard Weinberger wrote: > Hi! >=20 > Am 18.04.2015 um 16:58 schrieb Tom Marshall: > > On Sat, Apr 18, 2015 at 01:41:09PM +0200, Richard Weinberger wrote: > >> On Sat, Apr 18, 2015 at 12:20 AM, Tom Marshall wrote: > >>> So, I wrote a thing called 'zfile' that hooks into the VFS layer and > >>> intercepts file_operations to do file (de)compression on the fly. Whe= n a > >>> file is opened, it reads and decompresses the data into memory. The = file > >>> may be read, written, and mmaped in the usual way. If the contents a= re > >>> changed, the data is compressed and written back. A working patch fo= r an > >>> older kernel version may be found at: > >>> http://review.cyanogenmod.org/#/c/95220/ > >> > >> So, I've extracted the patch from that website and gave a quick review. > >> > >> I'm pretty sure VFS folks will hate the VFS layering you do. > >=20 > > This, I'm afraid, is the biggest obstacle to such a solution. I know t= hat > > OverlayFS has been merged, so filesystem stacking is acceptable. Perha= ps > > there would be a way to design a filesystem that stacks compression? >=20 > That's why I said think of adding compression support to ecryptfs. I think adding compression support to eCryptfs is the wrong approach. The "X is already a stacked filesystem so look into adding compression support to it" logic also works when X=3Doverlayfs. I doubt that Miklos would be willing to accept such a feature. :) A stacked filesystem that implements compression should be fairly simple. If it is not simple, it is too complicated to try to wedge into an unrelated stacked filesystem. While it may be the quickest route to your end goal, it will overly complicate eCryptfs. eCryptfs already has plenty of complexity around the file offset since metadata may be stored in the first 8192 bytes of the lower file, which offsets the entire file, and the end of the file has to be padded. Mixing in compression would make things much worse. Also, eCryptfs has lots of cruft that is definitely not needed for compression. Please consider a small, standalone stacked fs to handle compression. Tyler >=20 > >> Beside of that you decompress the *whole* file into memory at open() t= ime. > >> This will explode as soon you deal with bigger files. > >=20 > > I was thinking that a header with compressed offsets might be an option= =2E Or > > in the case of lz4 it's not terribly inefficient to scan the blocks. > >=20 > >> Also you seem to trust the user.compression.realsize xattr provided by > >> userspace. That looks exploitable. > >=20 > > This is only used to provide a fast stat(). It could be put into a hea= der > > or even removed entirely in favor of scanning the blocks. > >=20 > >> Back to my original question, why not FUSE? > >=20 > > Mostly because I'm not very familiar with FUSE. But I suppose it could= be > > an option. I have some immediate concerns though: > >=20 > > * How would it affect performance? FUSE passes all operations through = user > > space, correct? >=20 > Yeah, but you'll have to do benchmarks to find out the real trade off. > I'd give it a tray. Your targets are smartphones not HPC clusters. >=20 > > * How big might a reasonably complete implementation be for ARM? The > > implementation would need to be stored in the initrd. >=20 > I bet you can do it in less than 1000 lines of C... >=20 > >> Or add compression support to ecryptfs... > >=20 > > Several filesystems have native compression support. But this would vi= olate > > the goal of not switching filesystems. >=20 > ...your goals. Be flexible. ;) > BTW: ecryptfs is an overlay filesystem. >=20 > Thanks, > //richard > -- > To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" = in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html --y0ulUmNC+osPPQO6 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: Digital signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQIcBAEBCgAGBQJVNRMEAAoJENaSAD2qAscKA64QAI7Iyi9oIE5IVTBkPNK1WTJd DfByASKO+5WQHfHQ5C5pzEbF0iaoYFZ0poi5tuZTEGROF5rS+6cbdWfc45D7gc1q SZvXZ1cq4jZt7PTBYuahCSW1Bh/E4WEImSa9lxBPoiMDFM0p+iUgg0gGTk4ytuuK 3kK1YsbhN7xVnYPXbejAq4Bv6A9kXh87Vk/Cy0MkmJovOm5M5g2QkmbNq6v+8d2M u25P5YUMm0FLGBbk3LJ9qNDnUeb8JPNoYl2utfQeH12ECoH5rrZhkolBtOmOYmvr Cq/vGfvaigYdKrzXVCE3bslpKI6+ZxUbI0lmYh/4Yzq6ZBmc3lOns9zqHMDIDXZ2 DAn2f3xTrU9wHh9BiO3RtCUsO5I17k4KU9959XqkHVwfe8vp+PzlrZY42u5ugs5g w+dbX0RrEmWj6HNpBtdEQjhdAz3dKi4P67DAV3vmLemOf4zVlpPuAsnU+fxkQeow THEv9uzsDu3Dobp/kwevLatK730GKRFQ7+DAmq6LLB3VGHFMf31FYk+YF5sKEKCk tiHDSN1rfrAjH4Dp5OpyR7R55tOZ/fY+W0Ki0PLXXXvyWRljCzCXktFwRHfdW+v4 jQyE0XtARJ58jpInwR/1pDtI0COzPRi9PgPrYrujhE6Zb6hn0tQVQ0xffcOUbbqD VbU7nSzEzLs0iTl7v7MG =DdKd -----END PGP SIGNATURE----- --y0ulUmNC+osPPQO6--