[PATCH v2] Document the underlying protocol used by shallow repositories and --depth commands.

[PATCH v2] Document the underlying protocol used by shallow repositories and --depth commands.

[Alex Neronskiy]

Explain the exchange that occurs between a client and server when
the client is requesting shallow history and/or is already using
a shallow repository.

Signed-off-by: Alex Neronskiy <zakmagnus@google.com>
---
 Documentation/technical/pack-protocol.txt |   88 ++++++++++++++++++++++-------
 1 files changed, 67 insertions(+), 21 deletions(-)

diff --git a/Documentation/technical/pack-protocol.txt
b/Documentation/technical/pack-protocol.txt
index 369f91d..8727d40 100644
--- a/Documentation/technical/pack-protocol.txt
+++ b/Documentation/technical/pack-protocol.txt
@@ -187,26 +187,28 @@ server determine what the minimal packfile necessary for
transport is.
 
 Once the client has the initial list of references that the server
 has, as well as the list of capabilities, it will begin telling the
-server what objects it wants and what objects it has, so the server
-can make a packfile that only contains the objects that the client needs.
-The client will also send a list of the capabilities it wants to be in
-effect, out of what the server said it could do with the first 'want' line.
+server what objects it wants, its shallow objects (if any), and the
+maximum commit depth it wants (if any).  The client will also send a
+list of the capabilities it wants to be in effect, out of what the
+server said it could do with the first 'want' line.
 
 ----
   upload-request    =  want-list
-		       have-list
-		       compute-end
+		       *shallow-line
+		       *1depth-request
+		       flush-pkt
 
   want-list         =  first-want
 		       *additional-want
-		       flush-pkt
+
+  shallow-line      =  PKT_LINE("shallow" SP obj-id)
+
+  depth-request     =  PKT_LINE("deepen" SP depth)
 
   first-want        =  PKT-LINE("want" SP obj-id SP capability-list LF)
   additional-want   =  PKT-LINE("want" SP obj-id LF)
 
-  have-list         =  *have-line
-  have-line         =  PKT-LINE("have" SP obj-id LF)
-  compute-end       =  flush-pkt / PKT-LINE("done")
+  depth             =  1*DIGIT
 ----
 
 Clients MUST send all the obj-ids it wants from the reference
@@ -215,21 +217,65 @@ discovery phase as 'want' lines. Clients MUST send at
least one
 obj-id in a 'want' command which did not appear in the response
 obtained through ref discovery.
 
-If client is requesting a shallow clone, it will now send a 'deepen'
-line with the depth it is requesting.
-
-Once all the "want"s (and optional 'deepen') are transferred,
-clients MUST send a flush-pkt. If the client has all the references
+The client MUST write all obj-ids which it only has shallow copies
+of (meaning that it does not have the parents of a commit) as
+'shallow' lines so that the server is aware of the limitations of
+the client's history. Clients MUST NOT mention an obj-id which
+it does not know exists on the server.
+
+The client now sends the maximum commit history depth it wants for
+this transaction, which is the number of commits it wants from the
+tip of the history, if any, as a 'deepen' line.  A depth of 0 is the
+same as not making a depth request. The client does not want to receive
+any commits beyond this depth, nor objects needed only to complete
+those commits. Commits whose parents are not received as a result are
+defined as shallow and marked as such in the server. This information
+is sent back to the client in the next step.
+
+Once all the 'want's and 'shallow's (and optional 'deepen') are
+transferred, clients MUST send a flush-pkt. If the client did not
+request a shallow history and has all of the most recent references
 on the server, client flushes and disconnects.
 
-TODO: shallow/unshallow response and document the deepen command in the ABNF.
+Otherwise, if the client sent a positive depth request, the server
+will determine which commits will and will not be shallow and
+send this information to the client. If the client did not request
+a positive depth, this step is skipped.
+
+----
+  shallow-update   =  *shallow-line
+		      *unshallow-line
+		      flush-pkt
+
+  shallow-line     =  PKT-LINE("shallow" SP obj-id)
+
+  unshallow-line   =  PKT-LINE("unshallow" SP obj-id)
+----
+
+If the client has requested a positive depth, the server will compute
+the set of commits which are no deeper than the desired depth, starting
+at the client's wants. The server writes 'shallow' lines for each
+commit whose parents will not be sent as a result. The server writes
+an 'unshallow' line for each commit which the client has indicated is
+shallow, but is no longer shallow at the currently requested depth
+(that is, its parents will now be sent). The server MUST NOT mark
+as unshallow anything which the client has not indicated was shallow.
 
 Now the client will send a list of the obj-ids it has using 'have'
-lines.  In multi_ack mode, the canonical implementation will send up
-to 32 of these at a time, then will send a flush-pkt.  The canonical
-implementation will skip ahead and send the next 32 immediately,
-so that there is always a block of 32 "in-flight on the wire" at a
-time.
+lines, so the server can make a packfile that only contains the objects
+that the client needs. In multi_ack mode, the canonical implementation
+will send up to 32 of these at a time, then will send a flush-pkt. The
+canonical implementation will skip ahead and send the next 32 immediately,
+so that there is always a block of 32 "in-flight on the wire" at a time.
+
+----
+  upload-haves      =  have-list
+		       compute-end
+
+  have-list         =  *have-line
+  have-line         =  PKT-LINE("have" SP obj-id LF)
+  compute-end       =  flush-pkt / PKT-LINE("done")
+----
 
 If the server reads 'have' lines, it then will respond by ACKing any
 of the obj-ids the client said it had that the server also has. The
-- 
1.7.3.1

Re: [PATCH v2] Document the underlying protocol used by shallow repositories and --depth commands.

[Junio C Hamano]

Alex Neronskiy <zakmagnus@google.com> writes:

>X-Gmane-NNTP-Posting-Host: sea.gmane.org
>User-Agent: Loom/3.14 (http://gmane.org/)

It seems like you posted from the news interface at gmane, and the
annoying &nbsp; are gone now.  The patch is still linewrapped (see the
hunk header starting with "@@ -187,26"), but this one I can fix up and
actually take a look ;-).

> Explain the exchange that occurs between a client and server when
> the client is requesting shallow history and/or is already using
> a shallow repository.

Notice the use of word "shallow" here. It talks about the depth of the
history.

> +... Commits whose parents are not received as a result are
> +defined as shallow and marked as such in the server. This information
> +is sent back to the client in the next step.

As it is documented for the first time, we didn't have a formal
terminology for calling these commits and it is this document's
responsibility to come up with a good one. We have used "shallow clone"
and "shallow history", and I agree with the use of adjective in these
contexts, but I am not sure if it is a good idea to call the commits at
the boundary of a shallow history "shallow"---the following sentences do
not parse well at least for me:

    "This commit is shallow."
    "This commit is not shallow, and it is a direct child of that commit,
    which is shallow."
    "That commit does not exist in this repository because it is an
    ancestor of a shallow commit".

But it may be just me. Better wording ideas, anybody?

By the way, Dscho, the shallow extension was your invention 4 and half
years ago. I think the description in this version is mostly accurate
(modulo the part that talks about an early client termination after
"shallow" and "deepen" are sent), but I'd appreciate if you can comment on
it to improve.

Thanks.

Re: [PATCH v2] Document the underlying protocol used by shallow repositories and --depth commands.

[Alex Neronskiy]

Junio C Hamano <gitster <at> pobox.com> writes:
> It seems like you posted from the news interface at gmane, and the
> annoying &nbsp; are gone now.  The patch is still linewrapped (see the
> hunk header starting with "@@ -187,26"), but this one I can fix up and
> actually take a look .
So is the one starting with "@@ -215,21 +217,65". Sorry, I failed once again.
It's not cc'ed, either. I'm thinking maybe Scott Chacon also has comments, since
he seems to have written the file I'm modifying?

> As it is documented for the first time, we didn't have a formal
> terminology for calling these commits and it is this document's
> responsibility to come up with a good one. We have used "shallow clone"
> and "shallow history", and I agree with the use of adjective in these
> contexts, but I am not sure if it is a good idea to call the commits at
> the boundary of a shallow history "shallow"---the following sentences do
> not parse well at least for me:
> 
>     "This commit is shallow."
>     "This commit is not shallow, and it is a direct child of that commit,
>     which is shallow."
>     "That commit does not exist in this repository because it is an
>     ancestor of a shallow commit".
> 
> But it may be just me. Better wording ideas, anybody?
Yes, I see... I just transplanted the terminology that the protocol uses. When
"shallow SHA" is written, it means something is at the history boundary. Then
again, when "deepen d" is written, it specifies a depth and not an amount to
increase depth by, so maybe the strings the protocol uses are just bad for human
comprehension.

Re: [PATCH v2] Document the underlying protocol used by shallow repositories and --depth commands.

[Johannes Schindelin]

Dear Junio,

On Tue, 7 Jun 2011, Junio C Hamano wrote:

> Alex Neronskiy <zakmagnus@google.com> writes:
> 
> By the way, Dscho, the shallow extension was your invention 4 and half
> years ago.

Yep, I remember. My iBook died in the process. But I got the job.

> I think the description in this version is mostly accurate (modulo the 
> part that talks about an early client termination after "shallow" and 
> "deepen" are sent), but I'd appreciate if you can comment on it to 
> improve.

I will, once I am back to normal after traveling. Unfortunately (for my 
Git contributions at least), I will be heavily traveling in the next time, 
maybe for the next two years. But as for the patch you requested my 
look-over, I expect to have enough time tomorrow, latest coming weekend.

Ciao,
Johannes