Re: [ANNOUNCE] Example Cogito Addon - cogito-bundle

[Shawn Pearce <spearce@spearce.org>]

Linus Torvalds <torvalds@osdl.org> wrote:
> On Wed, 18 Oct 2006, Shawn Pearce wrote:
> > 
> > Actually there is a point to storing thin packs.  When I pull from
> > a remote repo (or push to a remote repo) a huge number of objects
> > and the target disk that is about to receive that huge number of
> > loose objects is slooooooooow I would rather just store the thin
> > pack then store the loose objects.
> > 
> > Ideally that thin pack would be repacked (along with the other
> > existing packs) as quickly as possible into a self-contained pack.
> > But that of course is unlikely to happen in practice; especially
> > on a push.
> 
> I'm really nervous about keeping thin packs around. 
> 
> But a possibly good (and fairly simple) alternative would be to just 
> create a non-thin pack on the receiving side. Right now we unpack into a 
> lot of loose objects, but it should be possible to instead "unpack" into a 
> non-thin pack.
> 
> In other words, we could easily still use the thin pack for communication, 
> we'd just "fill it out" on the receiving side.

Funny, I had the same thought.  :-)

We already know how many objects are coming in on a thin pack;
its right there in the header.  We could just have some threshold
at which we start writing a full pack rather than unpacking.

Writing such a full pack would be a simple matter of copying the
input stream out to a temporary pack, but sticking any delta bases
into a table in memory.  At the end of the data stream if we have any
delta bases which weren't actually in that pack then find them and
copy them onto the end, update the header and recompute the checksum.
git-fastimport does some of that already, though its trivial code...

Worst case scenario would be the incoming thin pack is 100% deltas
as we would need to copy in a base object for every object mentioned
in the pack.

-- 
Shawn.