From mboxrd@z Thu Jan 1 00:00:00 1970 From: Tom Marshall Subject: Re: fs compression Date: Wed, 20 May 2015 21:28:19 -0700 Message-ID: <20150521042819.GA14709@eden.sea.cyngn.com> References: <1431145253-2019-1-git-send-email-jaegeuk@kernel.org> <1431145253-2019-3-git-send-email-jaegeuk@kernel.org> <20150513020208.GK15721@dastard> <20150513064802.GA48682@jaegeuk-mac02.hsd1.ca.comcast.net> <20150514003721.GN15721@dastard> <20150516132403.GA2998@thunk.org> <20150516171326.GA24795@eden.sea.cyngn.com> <20150520174635.GA17651@eden.sea.cyngn.com> <20150520213641.GM2871@thunk.org> <20150520224630.GA10927@eden.sea.cyngn.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Cc: Jaegeuk Kim , linux-fsdevel@vger.kernel.org To: Theodore Ts'o Return-path: Received: from mail-pd0-f180.google.com ([209.85.192.180]:35467 "EHLO mail-pd0-f180.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750917AbbEUE2W (ORCPT ); Thu, 21 May 2015 00:28:22 -0400 Received: by pdea3 with SMTP id a3so93867392pde.2 for ; Wed, 20 May 2015 21:28:22 -0700 (PDT) Content-Disposition: inline In-Reply-To: <20150520224630.GA10927@eden.sea.cyngn.com> Sender: linux-fsdevel-owner@vger.kernel.org List-ID: On Wed, May 20, 2015 at 03:46:30PM -0700, Tom Marshall wrote: > So I have this all working as described. I haven't implemented readahead > yet (readpages) so it's slow. I'll be doing that next. > > The other thing to note is that since the uncompressed size is stored inside > the file data, stat() requires reading both the inode and the first page of > the file. That's not optimal, but I don't know if other generic out-of-band > solutions (such as xattrs) would be any better. I suppose it depends on > whether the xattr info is read in with the inode or not. > > Also, on the subject of file size, I'm currently swapping out the i_size for > the compressed i_size before calling down into the filesystem's readpage. > Yes that's a nasty hack that I'll need to address. > > On Wed, May 20, 2015 at 05:36:41PM -0400, Theodore Ts'o wrote: > > On Wed, May 20, 2015 at 10:46:35AM -0700, Tom Marshall wrote: > > > So I've been playing around a bit and I have a basic strategy laid out. > > > Please let me know if I'm on the right track. > > > > > > Compressed file attributes > > > ========================== > > > > > > The filesystem is responsible for detecting whether a file is compressed and > > > hooking into the compression lib. This may be done with an inode flag, > > > xattr, or any other applicable method. No other special attributes are > > > necessary. > > > > So I assume what you are implementing is read-only compression; that > > is, once the file is written, and the attribute set indicating that > > this is a compressed file, it is now immutable. > > That is TBD. Our use case right now only requires read-only, but I think > read-write would be a nice thing if it's not too convoluted. > > fallocate() is supported on the major filesystems, and I imagine the same > mechanisms could be used to provide rewriting of the "compression clusters". It just occurred to me that this could be the answer to both a consistent i_size and providing read-write access. If compression clusters were implemented by holes in the underlying file and the header+blockmap were not stored in the file proper, then the compressed and uncompressed file sizes would match. Further, rewriting a compression cluster would be a simple matter of adjusting the allocated compression cluster blocks as fallocate() provides. I honestly don't have a clue how to storing the header+blockmap outside the file yet, but if we could, that would seem to tie everything together quite nicely. Thoughts?