Re: [PATCH 09/11] streaming: read non-delta incrementally from a pack

[Shawn Pearce <spearce@spearce.org>]

On Sun, May 15, 2011 at 17:30, Junio C Hamano <gitster@pobox.com> wrote:
> +static read_method_decl(pack_non_delta)

I am not a huge fan of these decl macros... but I can see how writing
out the same function prototype 3 times is annoying.

> +       switch (in_pack_type) {
> +       default:
> +               return -1; /* we do not do deltas for now */

Haha. Deltas are going to be painful. Very, very painful.

We actually try to stream deltas in JGit. Our implementation isn't
useful. It only works if the base object is only accessed in
sequential order by the delta instruction sequences.

I had plans to add an external delta base cache to
$GIT_DIR/objects/delta-base-cache using a block file format that is
random accessible, but has CRC-32 checksums on each block to still
ensure there isn't silent data corruption on reads. You can read more
about it here http://egit.eclipse.org/r/1724 but the patch is probably
stalled and will get abandoned.

I think the better strategy is to avoid delta compression altogether
for objects that are so big we cannot materialize them as a contiguous
buffer. What a reasonable limit is, I don't know... but its probably
got to be around 25-50 MB. The Android project (as an example) has 6+
MB XML documents in their source code repository that are very delta
compressible.

-- 
Shawn.