Git network protocol

Git network protocol

[Josef "Jeff" Sipek]

Hello,

I'm trying to implement the git protocol, and I am having a bit of an issue
with the lack of information available about it (please correct me if I
missed some source of information.)

I understand the basic format of the protocol, however I'm not sure what
"command" can follow what. I also noticed some odd inconsistencies (or maybe
I just don't see the pattern yet.) For example, a git-clone generates this
traffic:

C: git-upload-pack ....
S: SHA1 HEAD....
S: SHA1 refs/heads/master
S: flush
C: want SHA1...
C: want SHA1... (it wants the same SHA1 twice!)
C: flush
C: done
S: NAK
S: the pack...

Then, when it is time to git-fetch a new commit, I get:

C: git-upload-pack ....
S: SHA1 HEAD....
S: SHA1 refs/heads/master
S: flush
C: want SHA1...
C: flush
C: have SHA1
C: done
S: ACK SHA1 continue
S: ACK SHA1 (same hash)
S: the pack...

Then, if I try to git-fetch but there is nothing new, I get:

C: git-upload-pack ....
S: SHA1 HEAD....
S: SHA1 refs/heads/master
S: flush
C: flush
<client closes connection>

So, I can _assume_ that "done" tells the server that it is time to make a
pack. Why does the server use NAK during the clone operation, but ACK
during fetch? Why does the server ACK the same SHA1 twice? And why does the
client "want" the same SHA1 twice? It just seems odd.

I think it would be great if there was some kind of description somewhere
that detailed the protocol. Also, the daemon source isn't the prettiest
thing in the world.

Thanks,
Josef "Jeff" Sipek.

-- 
Reality is merely an illusion, albeit a very persistent one.
		- Albert Einstein

Re: Git network protocol

[Junio C Hamano]

jeffpc@josefsipek.net (Josef "Jeff" Sipek) writes:

> I'm trying to implement the git protocol, and I am having a bit of an issue
> with the lack of information available about it (please correct me if I
> missed some source of information.)

Documentation/technical/pack-protocol.txt, and "git show 1bd8c8f0".

> So, I can _assume_ that "done" tells the server that it is time to make a
> pack. Why does the server use NAK during the clone operation, but ACK
> during fetch? Why does the server ACK the same SHA1 twice? And why does the
> client "want" the same SHA1 twice? It just seems odd.

During the initial "SHA-1 name"/"want" exchange, the server and
the client negotiate the protocol extension (the document needs
to be updated at least for "multi-ack" extension).

After that, the client and the server try to determine what
commits the client has that are recent ancestors of "want"
commits.  This exchange is done by client sending bunch of
"have" to the server.  The server responds "ACK" (in the
original protocol) to say "I've seen enough and know a common
ancestor to use".  In multi-ack protocol (which is used in
modern git, v0.99.9 and later), the server can respond "ACK
continue" to say "I've seen enough on that branch so do not
bother sending 'have's for its ancestors, but do keep sending
from other branch."  If the server does not feel it saw enough,
it does not send either.

The client can send "flush" -- this asks the server to give NAK
if more "have"s need to be sent.  If there is no more "have"s to
be sent, "done" is sent.

The protocol streams; if you see an ACK from server it does not
usually mean it is ACKing the last 'have' client has sent.

Re: Git network protocol

[Josef "Jeff" Sipek]

On Sun, Aug 13, 2006 at 11:42:33PM -0700, Junio C Hamano wrote:
> jeffpc@josefsipek.net (Josef "Jeff" Sipek) writes:
> 
> > I'm trying to implement the git protocol, and I am having a bit of an issue
> > with the lack of information available about it (please correct me if I
> > missed some source of information.)
> 
> Documentation/technical/pack-protocol.txt

This is pretty much a vague example - or at least it feels vague if you
don't know the protocol. After reading the source, and your description, the
example makes a lot more sense.

> "git show 1bd8c8f0".

Neat.

> After that, the client and the server try to determine what
> commits the client has that are recent ancestors of "want"
> commits.  This exchange is done by client sending bunch of
> "have" to the server.  The server responds "ACK" (in the
> original protocol) to say "I've seen enough and know a common
> ancestor to use".  In multi-ack protocol (which is used in
> modern git, v0.99.9 and later), the server can respond "ACK
> continue" to say "I've seen enough on that branch so do not
> bother sending 'have's for its ancestors, but do keep sending
> from other branch."  If the server does not feel it saw enough,
> it does not send either.

So, if I understand this correctly, multi_ack allows for multiple branches
to be fetched using the same connection?

> The protocol streams; if you see an ACK from server it does not
> usually mean it is ACKing the last 'have' client has sent.

Yeah, makes sense. Doing things over lo does things like that - I'm not sure
why I didn't realize it earlier.

Thanks,
Josef "Jeff" Sipek.

-- 
Don't drink and derive. Alcohol and algebra don't mix.

Re: Git network protocol

[Junio C Hamano]

jeffpc@josefsipek.net (Josef "Jeff" Sipek) writes:

> So, if I understand this correctly, multi_ack allows for multiple branches
> to be fetched using the same connection?

The original protocol without extension already allowed it.
Suppose the global history was like this:

       o-----------------------o---o---o---o---o---o---o---c---c---c
      /
 o---o---o---o---o---o---o---o---s---s---s

where the server side had 's', the client side had 'c', and both
of them had 'o'.  The objective is to update client with three 's'
commits.

The exchange would go like this without multi-ack:

	S: SHA-1 name1 -- for the rightmost 'o' on the top branch
        S: SHA-2 name2 -- for the rightmost 's' on the bottom branch
	C: want SHA-2 -- ask for the second branch tip
	C: have SHA1 -- the rightmost 'c' on the top branch
	C: have SHA1 -- the parent of the above
	C: have SHA1 -- the parent of the above
        C: have SHA1 -- the parent of the above, rightmost 'o' on the top.
        C: ... more have from the top branch.
	S: ACK -- for the rightmost 'o' on the top branch

During this exchange, the server learns that the client and the
server shares the rightmost 'o' on the top branch, but does not
learn about all the 'o' commits on the bottom branch, so it ends
up sending everything from the fork point to complete three 's'
commits.

The multi-ack extension was invented by Johannes to improve this
exchange.  It changes the protocol to let the server ACK with
"ACK continue".  After that, the client is expected to stop
traversing the parents of ACK-continue'd commit -- so it has a
chance to send the rightmost 'o' on the bottom branch.  When the
server sees it, it again gives an ACK, and the client soon runs
out have's to send and says "done".  In this case, the server
would have the rightmost 'o' commits on both branches to work
out the minimum set of objects to complete three 's' commits
that are missing from the client.

Re: Git network protocol

[Josef "Jeff" Sipek]

On Mon, Aug 14, 2006 at 05:59:37PM -0700, Junio C Hamano wrote:
> jeffpc@josefsipek.net (Josef "Jeff" Sipek) writes:
> 
> > So, if I understand this correctly, multi_ack allows for multiple branches
> > to be fetched using the same connection?
> 
> The original protocol without extension already allowed it.
> Suppose the global history was like this:
...

Thanks. That helped a lot.

Josef "Jeff" Sipek.

-- 
Linux, n.:
  Generous programmers from around the world all join forces to help you
  shoot yourself in the foot for free. 

Re: Mercurial 0.4b vs git patchbomb benchmark

[Noel Maddy]

On Fri, Apr 29, 2005 at 12:28:41PM -0700, Tom Lord wrote:
> 
> Think of it this way:
> 
>   (a) Joe, the mainline maintainer, gets a trusted message containing
>       a diff.
> 
>   (b) Joe reads the diff, it makes great sense, he wants to merge.
> 
>   (c) Joe downloads a tree.  Supposedly that tree is the result of
>       applying this diff.   The tree, not the diff, is used for
>       merging.

Call me a naive git, but seems to me the "git way" is a little
different. It's tree-based rather than diff-based, and doesn't involve
passing diffs around, right?

This is the process I'd expect:

    (a)' Joe is notified of an update made to an external git tree

    (b)' Joe pulls tree from the external git tree (signed by external
         developer)

    (c)' Joe reviews the (git-generated) diffs from his current
	     (trusted) tree to the new (signed) tree. If they pass
         review, he merges the new versions into his tree, commits,
         and signs his tree.

The logical hole that you point out is assuming that the diff is passed
separately from the tree rather than being directly generated from the
current maintainer tree and the signed remote tree.

If the diff is generated from the two signed trees, I don't see a hole.

Or am I missing something?


-- 
The world's largest Internet database in the country.
					      -- Trading Times radio ad
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
Noel Maddy <noel@zhtwn.com>

Re: Mercurial 0.4b vs git patchbomb benchmark

[Tom Lord]

  > Call me a naive git, but seems to me the "git way" is a little
  > different. It's tree-based rather than diff-based, and doesn't involve
  > passing diffs around, right?

Isn't that a significant part of what I said?  Go back and read more
carefully, is my suggestion.

  > Or am I missing something?

Very much so.


-t

Re: Mercurial 0.4b vs git patchbomb benchmark

[Noel Maddy]

On Fri, Apr 29, 2005 at 12:54:19PM -0700, Tom Lord wrote:
> 
> 
>   > Call me a naive git, but seems to me the "git way" is a little
>   > different. It's tree-based rather than diff-based, and doesn't involve
>   > passing diffs around, right?
> 
> Isn't that a significant part of what I said?  Go back and read more
> carefully, is my suggestion.

I'm trying to understand you. Please bear with me, and point out what
I'm missing.

Your example had Joe reviewing a signed diff, and then applying changes
from a tree that "supposedly" had the diff applied correctly, but may
have been corrupted. If the tree was not an accurate representation of
applying the diff, then the changes Joe applied to his tree will be
different than those that he reviewed.

My example had Joe downloading a remote signed tree, reviewing the changes
locally between his own trusted tree and the remote tree, and then
applying them locally. Since the diffs are generated locally between the
two trees, Joe is always reviewing the exact changes that will be
applied to his tree.

Doesn't this deal with the logical hole that you were pointing out in
your example? Or am I seeing a different "logical hole" than you are?


-- 
A man who fears nothing is a man who loves nothing.  And if you love
nothing, what joy is there in your life?
					 -- King Arthur, "First Knight"
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
Noel Maddy <noel@zhtwn.com>

git network protocol

[David Lang]

would it make sense for the network git protocol to be something along the 
lines of

client contacts server and sends
the tag you want to sync with (defaults to head)
the local index file

then the server can use the git tools locally to figure out what objects 
need to be sent to do the merge and only send those objects.

no this isn't as efficiant as only sending diffs, but it avoids sending 
any objects that aren't needed (which would be sent if you just did a 
straight rsync)

David Lang

-- 
There are two ways of constructing a software design. One way is to make it so simple that there are obviously no deficiencies. And the other way is to make it so complicated that there are no obvious deficiencies.
  -- C.A.R. Hoare

Re: git network protocol

[Daniel Barkalow]

On Fri, 29 Apr 2005, David Lang wrote:

> would it make sense for the network git protocol to be something along the 
> lines of
> 
> client contacts server and sends
> the tag you want to sync with (defaults to head)
> the local index file

Actually, you really want to have a bidirectional interaction, where the
client first fetches the info to determine where to start, and then goes
through the reachable space, asking for anything it doesn't already have.

(In the long run, we want to keep track of some things we already have all
of, or know we're missing, etc., so the receiver side doesn't have to
look over its whole tree.)

git already includes two versions of this protocol; the first runs against
a static HTTP server, and the second uses ssh to get a socket. At some
point, I'm going to enable these programs to read and write
.git/refs/?/? to figure out what they're supposed to get.

	-Daniel
*This .sig left intentionally blank*