Re: Request for detailed documentation of git pack protocol

["Shawn O. Pearce" <spearce@spearce.org>]

Jakub Narebski <jnareb@gmail.com> wrote:
> We have now proliferation of different (re)implementations of git:
> JGit in Java, Dulwich in Python, Grit in Ruby; and there are other
> planned: git# / managed git in C# (GSoC Mono project), ObjectiveGit
> in Objective-C (for iPhone IIRC).  At some time they would reach
> the point (or reached it already) of implementing git-daemon...
> but currently the documentation of git protocol is lacking.

Well, lets see...

JGit - me and Robin, here on git ML.  JGit is the closest
reimplementation effort to the canonical C implementation.
JGit runs in production servers for many folks, e.g. quite
a few Google engineers use the JGit server every day.  Its
our main git daemon.

Grit - GitHub folks.  They know where to find us.  And their
business is Git.  If Grit isn't right, they'll make it right,
or possibly suffer a loss of customers.  I'm fairly certain
that GitHub runs Grit in production.

ObjectGit - Scott Chacon, again, a GitHub folk.  Though he has
expressed interest in moving to JGit or libgit2 where/when possible.

Dulwich - off in its own world and not even trying to match basic
protocol rules by just watching what happens when you telnet to a
git port.  No clue how that's going to fair.

git# - We'll see.  Mono GSoC Git projects have a really bad
reputation of ignoring the existing git knowledge and hoping
they can invent the wheel on their own.

> This can lead, as you can read from recent post on git mailing, to
> implementing details wrong (like Dulwich not using full SHA-1 where
> it should, leading to ordinary git clients to failing to fetch from it),
> or fail at best practices of implementation (like JGit last issue with
> deadlocking for multi_ack extension).

Dulwich is just busted.

No existing developers knew that the fetch-pack/upload-pack protocol
has this required implicit buffering consideration until JGit
deadlocked over it.  But even then I'm still not 100% sure this
is true, or if it is just an artifact of the JGit upload-pack side
implementation being partially wrong.

> The current documentation of git protocol is very sparse; the docs
> in Documentation/technical/pack-protocol.txt offer only a sketch of
> exchange.  You can find more, including pkt-line format, a way sideband
> is multiplexed, and how capabilities are negotiated between server and
> client in design document for "smart" HTTP server, for example in
>   Subject: Re: More on git over HTTP POST
>   Message-ID: <20080803025602.GB27465@spearce.org>
>   URL: http://thread.gmane.org/gmane.comp.version-control.git/91104/focus=91196

Seriously?  Don't link to that.  Its a horrible version of the smart
HTTP RFC, and worse, it doesn't describe what you say it describes.
 
> It would be really nice, I think, to have RFC for git pack protocol.
> And it would help avoid incompatibilities between different clients
> and servers.  If the document would contain expected behaviour of
> client and server and Best Current Practices it would help avoid
> pitfals when implementing git-daemon in other implementation.

Yea, it would be nice.  But find me someone who knows the protocol
and who has the time to document the #!@* thing.  Maybe I'll try
to work on this myself, but I'm strapped for time, especially over
the next two-to-three months.

And lets not even start to mention Dulwich not completing a thin
pack before storing it on disk.  Those sorts of on disk things
matter to other more popular Git implementations (c git, jgit).
 
-- 
Shawn.