Re: Request for detailed documentation of git pack protocol

[Jakub Narebski <jnareb@gmail.com>]

This is combined response to various messages in this thread, following
my discoveries done using simple Perl script (using IO::Socket) which
assumes role of a git client, tested against github.com (IIRC it uses
Ruby implementation) and git.kernel.org (C Git), and "nc -l 9418".

By the way, is there some publicly accessible JGit (Java) and Dulwich
(Python) git-daemon one can test against?

  sp = Shawn O. Pearce
  jn = Jakub Narebski
  gb = Git Community Book (http://book.git-scm.com)


jn>> I meant that in the request line for fetching via git:// protocol
jn>>
jn>>       0032git-upload-pack /project.git\\000host=myserver.com\\000
jn>>
jn>> you separate path to repository from extra options using "\0" / NUL
jn>> as a separator. Well, this is only sane separator, as it is path
jn>> terminator, the only character which cannot appear in pathname
jn>> (although I do wonder whether project names with e.g. control
jn>> characters or UTF-8 characters would work correctly).
sp>
sp> No, that isn't the reason '\0' is used here.  But yea, that is true.
sp>
sp> The reason \0 is used is, git-daemon reads the 4 byte length, decodes
sp> that, then reads that many bytes.  Finally it writes a '\0' at the
sp> end of what it read, so that the entire "line" is NUL terminated.
sp> Then it reads the "command path" part from the resulting C string.
sp>
sp> The host=myserver.com part came later, after many daemons were
sp> already running all over the world.  By hiding it behind the '\0'
sp> an old daemon would never see it (but strlen() returned a value that
sp> was less than the length read, but the old daemons didn't care).
sp> Newer daemons look for where strlen() < length, and assume that
sp> the host header follows.
sp>
sp> The host header ends with '\0' in case additional headers would
sp> also appear here in the future.  IOW, like HTTP allows new headers
sp> to be added before the "\r\n\r\n" terminator at the body, we allow
sp> them between "\0".
[...]

sp> The NUL at the end of the host name is not strictly required, but
sp> must be present if the client were to ever pass additional options
sp> to the server.

Actually both git.kernel.org and github.com failed (deadlocked / hung)
when I tried to add extra key=value parameter at the end of request:

  003bgit-upload-pack /project.git\0host=myserver.com\0user=me\0

Hmmmm...


jn>> Hmmm... the communication between server and client is not entirely
jn>> clean. Do I understand correctly that this NAK is response to
jn>> clients flush after all those "want" lines?
sp>
sp> Yes.
sp>
jn>> And that "0009done" from client
jn>> tells server that it should send everything it has?
sp>
sp> Yes.  It means the client will not issue any more "have" lines,
sp> as it has nothing further in its history, so the server just has
sp> to give up and start generating a pack based on what it knows.

Here we were talking about the following part of exchange: 
(I have added "C:" prefix to signal that this is what client, 
git-clone here, sends; I have added also explicit "\n" to mark LF
characters terminating lines, and put each pkt-line on separate line)

gb>  C: 0054want 74730d410fcb6603ace96f1dc55ea6196122532d multi_ack side-band-64k ofs-delta\n
gb>  C: 0032want 7d1665144a3a975c05f1f43902ddaf084e784dbe\n
gb>  C: 0032want 5a3f6be755bbb7deae50065988cbfa1ffa9ab68a\n
gb>  C: 0032want 7e47fe2bd8d01d481f44d7af0531bd93d3b21c01\n
gb>  C: 0032want 74730d410fcb6603ace96f1dc55ea6196122532d\n
gb>  C: 0000
gb>  C: 0009done\n

and where server response is (again the quote from "Git Community Book"
was modified, removing here doublequotes and doubling of backslashes):

gb>  S: 0008NAK\n
gb>  S: 0023\002Counting objects: 2797, done.\n
gb>  [...]
gb>  S: 2004\001PACK\000\000\000\002 [...]

I have thought that after sending "0000" flush line client can wait for
NAK or ACK server response... but it is not the case.  When I tried to
read from server after "0000" flush and before "0009done\n", my client
(or netcat instance) deadlocked (hung) waiting for server response.
I either did a mistake in my fake client, or I don't understand git pack
protocol correctly.  Should client wait for NAK or ACK from server _only_
after sending maximum number of want/have lines (256 if I remember 
correctly?)?

When I removed sending "0000" flush line my fake client again hung 
(deadlocked?) waiting for server.


jn>> P.S. By the way, is pkt-line format original invention, or was it 
jn>> 'borrowed' from some other standard or protocol?
sp>
sp> No clue.  I find it f'king odd that the length is in hex.  There
sp> isn't much value to the protocol being human readable.  The PACK
sp> part of the stream sure as hell ain't.  You aren't going to type
sp> out a sequence of "have" lines against the remote, like you could
sp> with say an HTTP GET.  *shrug*

"git gui blame pkt-line.c" shows that pkt-line format is Linus invention.

It looks quite a bit like 'chunked' transfer encoding[1] in HTTP; there
each non-empty chunk starts with the number of octets of the data it
embeds (size written in hexadecimal) followed by a CRLF (carriage return
and linefeed), and the data itself. The chunk is then closed with a CRLF.
In some implementations, white space chars (0x20) are padded between
chunk-size and the CRLF.  In pkt-line format number of octet has fixed
width (4 hexadecimal digits, 0-padded), and we do not use CRLF as 
terminator of chunk/packet length and of chunk/packet itself.

In HTTP 'chunked' transfer encoding the last chunk is a single line,
simply made of the chunk-size (0).  In pkt-line format we use special
size of "0000" for a flush packet.

[1] http://en.wikipedia.org/wiki/Chunked_transfer_encoding

-- 
Jakub Narebski
Poland