Re: GTP/0.1 terminology 101: commit reels and references

[Sam Vilain <sam@vilain.net>]

On Mon, 2008-07-28 at 00:24 -0700, Junio C Hamano wrote:
> >>   Commit reels can also, and generally do, include the objects required
> >>  for a specific commit.
> >
> > Yes.  The only times where they wouldn't contain all the objects
> > required for the commits within the reel, is when those objects happened
> > to be contained by a previous reel.
> 
> What do you mean by "previous" reel?  It is not quite defined in your
> message but perhaps defined elsewhere?
> 
> How is this different from a bundle?  Does a reel, unlike a bundle,
> contain the full tree for the bottom commits? 

They are almost identical, both being defined by a set of starting and
ending refs.  And now that you mention it, I feel slightly embarrassed
for not spotting the connection before.  I only really compared reels to
packs, which is what the original specification tried to chop up bitwise
and distribute chunk by chunk.

The differences are:

  - the reel has a defined object order (which as I hoped to demonstrate
    in the test cases, is just a refinement of rev-list --date-order)

  - deltas always point in one direction, to objects "earlier" on
    the reel, so that slices of the reel sent on the network can be made
    thin without resulting in unresolvable deltas (which should be
    possible to do on commit boundaries using rev-list --objects-edge)

  - the behaviour at the beginning of the reel is precisely defined
    (although as I said, I think that the decision might be worth
    revisiting - perhaps getting just the latest reel is a useful
    'shallow clone')

> > This is one of the design decisions which I think may be a mistake; a
> > less expensive to calculate definition of a reel would be *all* objects
> > between the starting and ending Reference objects.
> 
> Do you mean all such objects and nothing else?  That would imply that a
> reel is quite similar to a bundle (but neither rev-list --objects-edge
> nor bundle guarantees that the result is minimal).

It's the lack of guarantees which is the issue, really.  In order to
take the download work of the entire pack and distribute it over
multiple peers, you need a way to carve the bundle up.  This has to
happen in such a way that the fragments that you get back will actually
fit together at the end, and also in such a way that you don't lose the
benefits of delta compression.

The way I thought would be best to do that would be to line up all the
objects in an exactly defined way - hence, the "reel" concept - and then
chop that up.

If a pack is already arranged to line up with the commit reel's
structure, then it's possible that the amount of work required to answer
a "play" request is as little as looking up in the reel index the local
on-disk location within the local pack, and copying that to the network.

I've certainly wondered how much baggage could be removed from this
whole thing, like replacing the "tracker" with a simple git-daemon
message that holds a register of mirrors/peers, possibly layering things
over git:// instead of the bittorrent-like protocol, dividing up blocks
by the commit graph and not all objects, etc.  But I think that it would
be best to defer that kind of design change until the conclusion of this
prototype experiment.

That being said, anything which does shortcut the distance to the finish
line and can be agreed on wouldn't go amiss.

Sam