Re: space compression (again)

["C. Scott Ananian" <cscott@cscott.net>]

On Fri, 15 Apr 2005, Linus Torvalds wrote:

> The problem with chunking is:
> - it complicates a lot of the routines. Things like "is this file
>   unchanged" suddenly become "is this file still the same set of chunks",
>   which is just a _lot_ more code and a lot more likely to have bugs.

The blob still has the same hash; therefore the file is still the same.
Nothing looks inside blobs; they just want either the hash or the full 
contents (if I understand the algorithms correctly).
I agree it's more code, but I think it can be nicely layered.

> - you have to find a blocking factor. I thought of just going it fixed
>   chunks, and that just doesn't help at all.

rsync uses a fixed chunk size, but this chunk can start at any offset (ie, 
not constrained to fixed boundaries).  This means that adding a single 
line to the file works like you'd expect, even though all the chunk 
boundaries change.  [I think this is what you're talking about.]

> - we already have wasted space due to the low-level filesystem (as
>   opposed to "git") usually being block-based, which means that space
>   utilization for small objects tends to suck. So you really want to
>   prefer objects that are several kB (compressed), and a small block just
>   wastes tons of space.

Not on (say) reiserfs, and not over the network.  I'm proposing (at the 
moment) easy conversion from chunked to unchunked disk representation,
so that you can leave things unchunked if (for example) you know you're 
running ext2 with a large block size.

> - there _is_ a natural blocking factor already. That's what a file
>   boundary really is within the project, and finding any other is really
>   quite hard.

Well, yes, it may be nontrivial.  But 'quite hard' depends on your 
perspective, I guess.  Given a cache of existing chunks, it's just a 
few table lookups. =)

> So I'm personally 100% sure that it's not worth it. But I'm not opposed to
> the _concept_: it makes total sense in the "filesystem" view, and is 100%
> equivalent to having an inode with pointers to blocks. I just don't think
> the concept plays out well in reality.

So I guess I'll have to implement this and find out, won't I? =)
  --scott

AMLASH overthrow SDI Suharto HBDRILL SMOTH SUMAC SYNCARP kibo Blair 
Diplomat Kojarena CIA cracking counter-intelligence CABOUNCE anthrax
                          ( http://cscott.net/ )