[PATCH] Document smart http

[PATCH] Document smart http

[Nguyễn Thái Ngọc Duy]

This may provide some clues for those who want to modify smart http
code as smart http is pretty much undocumented. Smart http "document"
so far is a few commit messages and the source code.

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 Documentation/git-fetch-pack.txt          | 11 +++--
 Documentation/git-receive-pack.txt        | 12 +++++-
 Documentation/git-send-pack.txt           |  9 +++-
 Documentation/git-upload-pack.txt         |  9 +++-
 Documentation/technical/pack-protocol.txt | 69 +++++++++++++++++++++++++++++++
 5 files changed, 103 insertions(+), 7 deletions(-)

diff --git a/Documentation/git-fetch-pack.txt b/Documentation/git-fetch-pack.txt
index 1e71754..85a9437 100644
--- a/Documentation/git-fetch-pack.txt
+++ b/Documentation/git-fetch-pack.txt
@@ -9,10 +9,7 @@ git-fetch-pack - Receive missing objects from another repository
 SYNOPSIS
 --------
 [verse]
-'git fetch-pack' [--all] [--quiet|-q] [--keep|-k] [--thin] [--include-tag]
-	[--upload-pack=<git-upload-pack>]
-	[--depth=<n>] [--no-progress]
-	[-v] [<host>:]<directory> [<refs>...]
+'git fetch-pack' [options] [<host>:]<directory> [<refs>...]
 
 DESCRIPTION
 -----------
@@ -90,6 +87,12 @@ be in a separate packet, and the list must end with a flush packet.
 --no-progress::
 	Do not show the progress.
 
+--stateless-rpc::
+	Smart HTTP mode.
+
+--lock-pack::
+	Issue "lock" command to the remote helper via stdout.
+
 -v::
 	Run verbosely.
 
diff --git a/Documentation/git-receive-pack.txt b/Documentation/git-receive-pack.txt
index b1f7dc6..0b2029c 100644
--- a/Documentation/git-receive-pack.txt
+++ b/Documentation/git-receive-pack.txt
@@ -9,7 +9,7 @@ git-receive-pack - Receive what is pushed into the repository
 SYNOPSIS
 --------
 [verse]
-'git-receive-pack' <directory>
+'git-receive-pack' [options] <directory>
 
 DESCRIPTION
 -----------
@@ -35,6 +35,16 @@ are not fast-forwards.
 
 OPTIONS
 -------
+--stateless-rpc::
+	Smart HTTP mode.
+
+--advertise-refs::
+	Only the initial ref advertisement is output then exits
+	immediately.
+
+--quiet::
+	Make unpack-objects at the receive-pack end quiet.
+
 <directory>::
 	The repository to sync into.
 
diff --git a/Documentation/git-send-pack.txt b/Documentation/git-send-pack.txt
index dc3a568..6cee3d4 100644
--- a/Documentation/git-send-pack.txt
+++ b/Documentation/git-send-pack.txt
@@ -9,7 +9,7 @@ git-send-pack - Push objects over Git protocol to another repository
 SYNOPSIS
 --------
 [verse]
-'git send-pack' [--all] [--dry-run] [--force] [--receive-pack=<git-receive-pack>] [--verbose] [--thin] [<host>:]<directory> [<ref>...]
+'git send-pack' [options] [<host>:]<directory> [<ref>...]
 
 DESCRIPTION
 -----------
@@ -52,6 +52,13 @@ OPTIONS
 	Send a "thin" pack, which records objects in deltified form based
 	on objects not included in the pack to reduce network traffic.
 
+--stateless-rpc::
+	Smart HTTP mode.
+
+--helper-status:
+	Issue status commands (e.g. "ok" or "error") to the remote
+	helper via stdout.
+
 <host>::
 	A remote host to house the repository.  When this
 	part is specified, 'git-receive-pack' is invoked via
diff --git a/Documentation/git-upload-pack.txt b/Documentation/git-upload-pack.txt
index 0abc806..ce9a455 100644
--- a/Documentation/git-upload-pack.txt
+++ b/Documentation/git-upload-pack.txt
@@ -9,7 +9,7 @@ git-upload-pack - Send objects packed back to git-fetch-pack
 SYNOPSIS
 --------
 [verse]
-'git-upload-pack' [--strict] [--timeout=<n>] <directory>
+'git-upload-pack' [options] <directory>
 
 DESCRIPTION
 -----------
@@ -31,6 +31,13 @@ OPTIONS
 --timeout=<n>::
 	Interrupt transfer after <n> seconds of inactivity.
 
+--stateless-rpc::
+	Smart HTTP mode.
+
+--advertise-refs::
+	Only the initial ref advertisement is output then exits
+	immediately.
+
 <directory>::
 	The repository to sync from.
 
diff --git a/Documentation/technical/pack-protocol.txt b/Documentation/technical/pack-protocol.txt
index b898e97..7590394 100644
--- a/Documentation/technical/pack-protocol.txt
+++ b/Documentation/technical/pack-protocol.txt
@@ -546,3 +546,72 @@ An example client/server communication might look like this:
    S: 0018ok refs/heads/debug\n
    S: 002ang refs/heads/master non-fast-forward\n
 ----
+
+Smart HTTP Transport
+--------------------
+
+Smart HTTP protocol is basically "git protocol on top of http". The
+base protocol is modified slightly to fit HTTP processing model: no
+bidirectional full-duplex connections, the program may read the
+request, write a response and must exit. Any negotiation is broken
+down into many separate GET or POST requests. The server is
+stateless. The client must send enough information so that the server
+can recreate the previous state.
+
+Reference Discovery
+-------------------
+
+The server end always sends the list of references in both push and
+fetch cases. This ref list is retrieved by the client's sending HTTP
+GET request to a smart http url ending with
+"/info/refs?service=<service>" where <service> could be either
+git-upload-pack or git-receive-pack for fetching or pushing
+respectively. The output is in pkt-line format.
+
+----
+  advertised-refs  =  service
+		      flush-pkt
+		      (no-refs / list-of-refs)
+		      flush-pkt
+
+  service          =  PKT-LINE("# service=" service-name)
+  service-name     =  ("git-upload-pack" / "git-receive-pack")
+
+  no-refs          =  PKT-LINE(zero-id SP "capabilities^{}"
+		      NUL capability-list LF)
+
+  list-of-refs     =  first-ref *other-ref
+  first-ref        =  PKT-LINE(obj-id SP refname
+		      NUL capability-list LF)
+
+  other-ref        =  PKT-LINE(other-tip / other-peeled)
+  other-tip        =  obj-id SP refname LF
+  other-peeled     =  obj-id SP refname "^{}" LF
+
+  capability-list  =  capability *(SP capability)
+  capability       =  1*(LC_ALPHA / DIGIT / "-" / "_")
+  LC_ALPHA         =  %x61-7A
+----
+
+Packfile Negotiation
+--------------------
+
+For fetching, packet negotiation is via a series of HTTP POST requests
+to an url ending with "/git-upload-pack" with the content in pkt-line
+format. git-upload-pack's response consists of a "service" line like
+in Reference Discovery followed by normal git-upload-pack packet
+lines. Capability multi_ack_detailed is required by Smart HTTP.
+
+Common objects that are discovered are appended onto the request as
+have lines and are sent again on the next request. This allows the
+remote side to reinitialize its in-memory list of common objects
+during the next request and the remote does not need to maintain the
+negotiation state.
+
+Reference Update Request
+------------------------
+
+For pushing, a HTTP POST request is sent to an url ending with
+"/git-receive-pack" with the content in pkt-line format.
+git-receive-pack's response consists of a "service" line like in
+Reference Discovery followed by normal git-receive-pack packet lines.
-- 
1.8.2.82.gc24b958

Re: [PATCH] Document smart http

[Jeff King]

On Tue, Aug 20, 2013 at 12:08:08PM +0700, Nguyen Thai Ngoc Duy wrote:

> This may provide some clues for those who want to modify smart http
> code as smart http is pretty much undocumented. Smart http "document"
> so far is a few commit messages and the source code.

There was also this:

  http://article.gmane.org/gmane.comp.version-control.git/129734

which seems to have never gotten updated enough to be applied along with
the code. But with some updates to make sure it matches the current
behavior, it is probably a more comprehensive description.

But if you don't feel like spending more time on this on top of what
you've already done, I think the patch I'm responding to is better than
what we have now (i.e., nothing).

> +Reference Discovery
> +-------------------
> +
> +The server end always sends the list of references in both push and
> +fetch cases. This ref list is retrieved by the client's sending HTTP
> +GET request to a smart http url ending with
> +"/info/refs?service=<service>" where <service> could be either
> +git-upload-pack or git-receive-pack for fetching or pushing
> +respectively. The output is in pkt-line format.
> +
> +----
> +  advertised-refs  =  service
> +		      flush-pkt
> +		      (no-refs / list-of-refs)
> +		      flush-pkt
> +
> +  service          =  PKT-LINE("# service=" service-name)
> +  service-name     =  ("git-upload-pack" / "git-receive-pack")
> +
> +  no-refs          =  PKT-LINE(zero-id SP "capabilities^{}"
> +		      NUL capability-list LF)
> +
> +  list-of-refs     =  first-ref *other-ref
> +  first-ref        =  PKT-LINE(obj-id SP refname
> +		      NUL capability-list LF)
> +
> +  other-ref        =  PKT-LINE(other-tip / other-peeled)
> +  other-tip        =  obj-id SP refname LF
> +  other-peeled     =  obj-id SP refname "^{}" LF
> +
> +  capability-list  =  capability *(SP capability)
> +  capability       =  1*(LC_ALPHA / DIGIT / "-" / "_")
> +  LC_ALPHA         =  %x61-7A
> +----

Most of this is a repeat of what is in the earlier sections. I don't
think the protocol is changing much and these are not likely to get out of
date with each other, but I wonder if it is easier on the reader to
simply describe the output in terms of what is added on top of the
regular ref advertisement (i.e., the service line). Something like:

  stateless-advertised-refs =  service
                               advertised-refs

  service                   =  PKT-LINE("# service=" service-name)
  service-name              =  ("git-upload-pack" / "git-receive-pack")

where advertised-refs is defined in the earlier BNF. You may also want
to note:

  Servers may respond to a smart request with a regular `advertised-refs`
  response rather than a `stateless-advertised-refs` response. In this
  case, the client MUST assume that the server does not understand smart
  HTTP and either abort or proceed with the non-smart protocol.

-Peff

Re: [PATCH] Document smart http

[Duy Nguyen]

On Tue, Aug 20, 2013 at 9:16 PM, Jeff King <peff@peff.net> wrote:
> On Tue, Aug 20, 2013 at 12:08:08PM +0700, Nguyen Thai Ngoc Duy wrote:
>
>> This may provide some clues for those who want to modify smart http
>> code as smart http is pretty much undocumented. Smart http "document"
>> so far is a few commit messages and the source code.
>
> There was also this:
>
>   http://article.gmane.org/gmane.comp.version-control.git/129734
>
> which seems to have never gotten updated enough to be applied along with
> the code. But with some updates to make sure it matches the current
> behavior, it is probably a more comprehensive description.

If I knew about this patch, it could have saved me a lot of time
reading remote-curl.c. Will check it with current code and resubmit an
update of this version instead.
-- 
Duy