Re: Transparent compression in the FS

[jw schultz <jw@pegasys.ws>]

On Thu, Oct 16, 2003 at 06:29:26PM +0200, Andrea Arcangeli wrote:
> Hi Jeff,
> 
> On Wed, Oct 15, 2003 at 11:13:27AM -0400, Jeff Garzik wrote:
> > Josh and others should take a look at Plan9's venti file storage method 
> > -- archival storage is a series of unordered blocks, all of which are 
> > indexed by the sha1 hash of their contents.  This magically coalesces 
> > all duplicate blocks by its very nature, including the loooooong runs of 
> > zeroes that you'll find in many filesystems.  I bet savings on "all 
> > bytes in this block are zero" are worth a bunch right there.
> 
> I had a few ideas on the above.
> 
> if the zero blocks are the problem, there's a tool called zum that nukes
> them and replaces them with holes. I use it sometime, example:

With the exception of NUL blocks i doubt there is much
savings at any level below the file.  That is, only a tiny
fraction of files that are not fully duplicate of one
another will share many aligned blocks.

Now detecting files that are duplicates and linking them in
some way might be a useful in a low-priority daemon.  But
the links created would have to be sure to preserve them as
seperate inodes so that overwrites break the loose link but
not the user-created hardlink.


-- 
________________________________________________________________
	J.W. Schultz            Pegasystems Technologies
	email address:		jw@pegasys.ws

		Remember Cernan and Schmitt