Re: Add Git-aware CGI for Git-aware smart HTTP transport

Re: Add Git-aware CGI for Git-aware smart HTTP transport

[Shawn O. Pearce]

[...added git ML back to thread...]

"H. Peter Anvin" <hpa@zytor.com> wrote:
> Shawn O. Pearce wrote:
>> The plan I've proposed requires wedging the CGI in between the HTTP
>> server and the repository files.  Which means older dumb clients
>> get data by forking off the CGI, rather than letting the HTTP server
>> stream the file itself.
>
> Yeah, that's quite a bit unfortunate, because it means some potentially  
> very expensive buffering in Apache.  That's one reason to do some kind  
> of redirection.

Hmm.  So what if the "smart" protocol used a redirect to the CGI
and the dumb protocol didn't use any redirects at all?  I say this
because I think the dumb protocol won't handle redirects well.
It will do them, but it would incur a redirect on every request
it makes.

So if we have the "smart" protocol perform detection by trying:

	C: HEAD /path/to/repository.git/git-http-backend HTTP/1.0

	S: HTTP/1.0 302 Found
	S: Location: /git-http/path/to/repository.git

Under Apache this server configuration can be easily handled by a
mod_rewrite regex:

	RewriteRule ^(/pub/scm/.*)/git-http-backend$ /git/$1 [R,L]
	ScriptAlias /git/ /path/to/git-http-backend/

Individual users could also install the git-http-backend CGI
right into their repository, in which case the CGI if invoked with
no PATH_INFO can do a redirect back to itself to indicate where
GIT_DIR is:

	C: HEAD /path/to/repository.git/git-http-backend HTTP/1.0

	S: HTTP/1.0 302 Found
	S: Location: /path/to/repository.git/git-http-backend/.

Individual operations can be selected by appending on the operation
name, so <Location ~ > style rules can be used to apply access
controls, such as:

	# Disallow push to any smart repository via ScriptAlias
	#
	<Location ~ ^/git/.*/receive-pack$>
		Order Deny,Allow
		Deny from all
	</Location>

	# Disallow push to any smart repository with CGI in tree.
	#
	<Location ~ .*/git-http-backend/./receive-pack$>
		Order Deny,Allow
		Deny from all
	</Location>

Setting this up on a server which doesn't have the power of mod_regex
available would be tricky, as you need to link the CGI into every
single repository you are serving.  I don't know (or use) many other
HTTP servers beyond Apache so I'm not sure if they can do this.

-- 
Shawn.

Re: Add Git-aware CGI for Git-aware smart HTTP transport

[H. Peter Anvin]

Shawn O. Pearce wrote:
> 
> Hmm.  So what if the "smart" protocol used a redirect to the CGI
> and the dumb protocol didn't use any redirects at all?  I say this
> because I think the dumb protocol won't handle redirects well.
> It will do them, but it would incur a redirect on every request
> it makes.
> 

That's preferrable anyway, in my opinion.

> Setting this up on a server which doesn't have the power of mod_regex
> available would be tricky, as you need to link the CGI into every
> single repository you are serving.  I don't know (or use) many other
> HTTP servers beyond Apache so I'm not sure if they can do this.

Many can, and even more can if we instead of git-http-backend had 
something which looked vaguely like a unique extension, like 
"backend.git-http"

	-hpa

Re: More on git over HTTP POST

[Shawn O. Pearce]

"Shawn O. Pearce" <spearce@spearce.org> wrote:
> "H. Peter Anvin" <hpa@zytor.com> wrote:
> > I have investigated a bit what it would take to support git protocol  
> > (smart transport) over HTTP POST transactions.
> 
> I have started to think about this more myself, not just for POST
> put also for some form of GET that can return an efficient pack,
> rather than making the client walk the object chains itself.
...
> HTTP POST is actually trivial if you don't want to support the new
> tell-me-more extension that was added to git-push.  Hell, I could
> write the CGI in a few minutes I think.  Its really just a small
> wrapper around git-receive-pack.

So I have this draft of how smart push might work.  Its slated
for the Documentation/technical directory.  Thus far I have only
written about push support, but Ilari on #git has some ideas about
how to do a smart fetch protocol.

Implementation wise in C git I think this is just a new C
program (git-http-backend?) that turns around and proxies
into git-receive-pack, at least for the push support.

What I don't know is how we could configure URI translation from
/path/to/repository.git received out of the $PATH_INFO in the
CGI environment to a physical directory.  Should we rely on the
server's $PATH_TRANSLATED?


Smart HTTP transfer protocols
=============================

Git supports two HTTP based transfer protocols.  A "dumb" protocol
which requires only a standard HTTP server on the server end of the
connection, and a "smart" protocol which requires a Git aware CGI
(or server module).  This document describes the "smart" protocol.

Authentication
--------------

Standard HTTP authentication is used, and must be configured and
enforced by the HTTP server software.

Chunked Transfer Encoding
-------------------------

For performance reasons the HTTP/1.1 chunked transfer encoding is
used frequently to transfer variable length objects.  This avoids
needing to produce large results in memory to compute the proper
content-length.

Detecting Smart Servers
-----------------------

HTTP clients can detect a smart Git-aware server by sending the
show-ref request (below) to the server.  If the response has a
status of 200 and the magic x-application/git-refs content type
then the server can be assumed to be a smart Git-aware server.

If any other response is received the client must assume dumb
protocol support, as the server did not correctly response to
the request.


Show Refs
---------

Obtains the available refs from the remote repository.  The response
is a sequence of git "packet lines", one per ref, and a final flush
packet line to indicate the end of stream.

	C: GET /path/to/repository.git?show-ref HTTP/1.0

	S: HTTP/1.1 200 OK
	S: Content-Type: x-application/git-refs
	S: Transfer-Encoding: chunked
	S:
	S: 62
	S: 003e95dcfa3633004da0049d3d0fa03f80589cbcaf31 refs/heads/maint
	S: 
	S: 63
	S: 003fd049f6c27a2244e12041955e262a404c7faba355 refs/heads/master
	S: 
	S: 59
	S: 003b2cb58b79488a98d2721cea644875a8dd0026b115 refs/heads/pu
	S: 
	S: 4
	S: 0000
	S: 0

Push Pack
---------

Uploads a pack and updates refs.  The start of the stream is the
commands to update the refs and the remainder of the stream is the
pack file itself.  See git-receive-pack and its network protocol
in pack-protocol.txt, as this is essentially the same.

	C: POST /path/to/repository.git?receive-pack HTTP/1.0
	C: Content-Type: x-application/git-receive-pack
	C: Transfer-Encoding: chunked
	C:
	C: 103
	C: 006395dcfa3633004da0049d3d0fa03f80589cbcaf31 d049f6c27a2244e12041955e262a404c7faba355 refs/heads/maint
	C: 4
	C: 0000
	C: 12
	C: PACK
	...
	C: 0

	S: HTTP/1.0 200 OK
	S: Content-type: x-application/git-status
	S: Transfer-Encoding: chunked
	S:
	S: ...<output of receive-pack>...



-- 
Shawn.

[RFC 1/2] Add backdoor options to receive-pack for use in Git-aware CGI

[Shawn O. Pearce]

The new --report-status flag forces the status report feature of
the push protocol to be enabled.  This can be useful in a CGI
program that implements the server side of a "smart" Git-aware
HTTP transport.  The CGI code can perform the selection of the
feature and ask receive-pack to enable it automatically.

The new --no-advertise-heads causes receive-pack to bypass its usual
display of known refs to the client, and instead immediately start
reading the commands and pack from stdin.  This is useful in a CGI
situation where we want to hand off all input to receive-pack.

Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
---
 receive-pack.c |   19 ++++++++++++++-----
 1 files changed, 14 insertions(+), 5 deletions(-)

diff --git a/receive-pack.c b/receive-pack.c
index d44c19e..512eae6 100644
--- a/receive-pack.c
+++ b/receive-pack.c
@@ -464,6 +464,7 @@ static int delete_only(struct command *cmd)
 
 int main(int argc, char **argv)
 {
+	int advertise_heads = 1;
 	int i;
 	char *dir = NULL;
 
@@ -472,7 +473,15 @@ int main(int argc, char **argv)
 		char *arg = *argv++;
 
 		if (*arg == '-') {
-			/* Do flag handling here */
+			if (!strcmp(arg, "--report-status")) {
+				report_status = 1;
+				continue;
+			}
+			if (!strcmp(arg, "--no-advertise-heads")) {
+				advertise_heads = 0;
+				continue;
+			}
+
 			usage(receive_pack_usage);
 		}
 		if (dir)
@@ -497,10 +506,10 @@ int main(int argc, char **argv)
 	else if (0 <= receive_unpack_limit)
 		unpack_limit = receive_unpack_limit;
 
-	write_head_info();
-
-	/* EOF */
-	packet_flush(1);
+	if (advertise_heads) {
+		write_head_info();
+		packet_flush(1);
+	}
 
 	read_head_info();
 	if (commands) {
-- 
1.6.0.rc1.221.g9ae23

[RFC 2/2] Add Git-aware CGI for Git-aware smart HTTP transport

[Shawn O. Pearce]

This CGI can be loaded into an Apache server using ScriptAlias,
such as with the following configuration:

  LoadModule cgi_module /usr/libexec/apache2/mod_cgi.so
  LoadModule alias_module /usr/libexec/apache2/mod_alias.so
  ScriptAlias /git/ /usr/libexec/git-core/git-http-backend/

Repositories are accessed via the translated PATH_INFO.

The CGI is backwards compatible with the dumb client, allowing the
client to detect the server's smarts by looking at the content-type
returned from "GET /repo.git/info/refs".  If the returned content
type is the magic application/x-git-refs type then the client can
assume the server is Git-aware.

Signed-off-by: Shawn O. Pearce <spearce@spearce.org>
---
 .gitignore                                |    1 +
 Documentation/technical/http-protocol.txt |   88 +++++++++
 Makefile                                  |    1 +
 http-backend.c                            |  302 +++++++++++++++++++++++++++++
 4 files changed, 392 insertions(+), 0 deletions(-)
 create mode 100644 Documentation/technical/http-protocol.txt
 create mode 100644 http-backend.c

diff --git a/.gitignore b/.gitignore
index a213e8e..02eaf3a 100644
--- a/.gitignore
+++ b/.gitignore
@@ -51,6 +51,7 @@ git-gc
 git-get-tar-commit-id
 git-grep
 git-hash-object
+git-http-backend
 git-http-fetch
 git-http-push
 git-imap-send
diff --git a/Documentation/technical/http-protocol.txt b/Documentation/technical/http-protocol.txt
new file mode 100644
index 0000000..6cb96f3
--- /dev/null
+++ b/Documentation/technical/http-protocol.txt
@@ -0,0 +1,88 @@
+Smart HTTP transfer protocols
+=============================
+
+Git supports two HTTP based transfer protocols.  A "dumb" protocol
+which requires only a standard HTTP server on the server end of the
+connection, and a "smart" protocol which requires a Git aware CGI
+(or server module).  This document describes the "smart" protocol.
+
+As a design feature smart servers automatically degrade to the
+dumb protocol when speaking with a dumb client.  This may cause
+more load to be placed on the server as the file GET requests are
+handled by a CGI rather than the server itself.
+
+
+Authentication
+--------------
+
+Standard HTTP authentication is used, and must be configured and
+enforced by the HTTP server software.
+
+Chunked Transfer Encoding
+-------------------------
+
+For performance reasons the HTTP/1.1 chunked transfer encoding is
+used frequently to transfer variable length objects.  This avoids
+needing to produce large results in memory to compute the proper
+content-length.
+
+Detecting Smart Servers
+-----------------------
+
+HTTP clients can detect a smart Git-aware server by sending the
+/info/refs request (below) to the server.  If the response has a
+status of 200 and the magic application/x-git-refs content type
+then the server can be assumed to be a smart Git-aware server.
+
+
+Show Refs
+---------
+
+Obtains the available refs from the remote repository.  The response
+is a sequence of refs, one per line.  The actual format matches that
+of the $GIT_DIR/info/refs file normally used by a "dumb" protocol.
+
+	C: GET /path/to/repository.git/info/refs HTTP/1.0
+
+	S: HTTP/1.1 200 OK
+	S: Content-Type: application/x-git-refs
+	S: Transfer-Encoding: chunked
+	S:
+	S: 62
+	S: 95dcfa3633004da0049d3d0fa03f80589cbcaf31 refs/heads/maint
+	S:
+	S: 63
+	S: d049f6c27a2244e12041955e262a404c7faba355 refs/heads/master
+	S:
+	S: 59
+	S: 2cb58b79488a98d2721cea644875a8dd0026b115 refs/heads/pu
+	S:
+
+Push Pack
+---------
+
+Uploads a pack and updates refs.  The start of the stream is the
+commands to update the refs and the remainder of the stream is the
+pack file itself.  See git-receive-pack and its network protocol
+in pack-protocol.txt, as this is essentially the same.
+
+	C: POST /path/to/repository.git/receive-pack HTTP/1.0
+	C: Content-Type: application/x-git-receive-pack
+	C: Transfer-Encoding: chunked
+	C:
+	C: 103
+	C: 006395dcfa3633004da0049d3d0fa03f80589cbcaf31 d049f6c27a2244e12041955e262a404c7faba355 refs/heads/maint
+	C: 4
+	C: 0000
+	C: 12
+	C: PACK
+	...
+	C: 0
+
+	S: HTTP/1.0 200 OK
+	S: Content-type: application/x-git-receive-pack-status
+	S: Transfer-Encoding: chunked
+	S:
+	S: ...<output of receive-pack>...
+
+
diff --git a/Makefile b/Makefile
index 52c67c1..3a93bf6 100644
--- a/Makefile
+++ b/Makefile
@@ -298,6 +298,7 @@ PROGRAMS += git-unpack-file$X
 PROGRAMS += git-update-server-info$X
 PROGRAMS += git-upload-pack$X
 PROGRAMS += git-var$X
+PROGRAMS += git-http-backend$X
 
 # List built-in command $C whose implementation cmd_$C() is not in
 # builtin-$C.o but is linked in as part of some other command.
diff --git a/http-backend.c b/http-backend.c
new file mode 100644
index 0000000..a498f89
--- /dev/null
+++ b/http-backend.c
@@ -0,0 +1,302 @@
+#include "cache.h"
+#include "refs.h"
+#include "pkt-line.h"
+#include "object.h"
+#include "tag.h"
+#include "exec_cmd.h"
+#include "run-command.h"
+
+static const char content_type[] = "Content-Type";
+static const char content_length[] = "Content-Length";
+
+static int can_chunk;
+static char buffer[1000];
+
+static void send_status(unsigned code, const char *msg)
+{
+	size_t n;
+
+	n = snprintf(buffer, sizeof(buffer), "Status: %u %s\r\n", code, msg);
+	if (n >= sizeof(buffer))
+		die("protocol error: impossibly long header");
+	safe_write(1, buffer, n);
+}
+
+static void send_header(const char *name, const char *value)
+{
+	size_t n;
+
+	n = snprintf(buffer, sizeof(buffer), "%s: %s\r\n", name, value);
+	if (n >= sizeof(buffer))
+		die("protocol error: impossibly long header");
+	safe_write(1, buffer, n);
+}
+
+static void end_headers(void)
+{
+	safe_write(1, "\r\n", 2);
+}
+
+static void send_nocaching(void)
+{
+	const char *proto = getenv("SERVER_PROTOCOL");
+	if (!proto || !strcmp(proto, "HTTP/1.0"))
+		send_header("Expires", "Mon, 17 Sep 2001 00:00:00 GMT");
+	else
+		send_header("Cache-Control", "no-cache");
+}
+
+static void send_connection_close(void)
+{
+	send_header("Connection", "close");
+}
+
+static void enable_chunking(void)
+{
+	const char *proto = getenv("SERVER_PROTOCOL");
+
+	can_chunk = proto && strcmp(proto, "HTTP/1.0");
+	if (can_chunk)
+		send_header("Transfer-Encoding", "chunked");
+	else
+		send_connection_close();
+}
+
+#define hex(a) (hexchar[(a) & 15])
+static void chunked_write(const char *fmt, ...)
+{
+	static const char hexchar[] = "0123456789abcdef";
+	va_list args;
+	unsigned n;
+
+	va_start(args, fmt);
+	n = vsnprintf(buffer + 6, sizeof(buffer) - 8, fmt, args);
+	va_end(args);
+	if (n >= sizeof(buffer) - 8)
+		die("protocol error: impossibly long line");
+
+	if (can_chunk) {
+		unsigned len = n + 4, b = 4;
+
+		buffer[4] = '\r';
+		buffer[5] = '\n';
+		buffer[n + 6] = '\r';
+		buffer[n + 7] = '\n';
+
+		while (n > 0) {
+			buffer[--b] = hex(n);
+			n >>= 4;
+			len++;
+		}
+
+		safe_write(1, buffer + b, len);
+	} else
+		safe_write(1, buffer + 6, n);
+}
+
+static void end_chunking(void)
+{
+	static const char flush_chunk[] = "0\r\n\r\n";
+	if (can_chunk)
+		safe_write(1, flush_chunk, strlen(flush_chunk));
+}
+
+static void NORETURN invalid_request(const char *msg)
+{
+	static const char header[] = "error: ";
+
+	send_status(400, "Bad Request");
+	send_header(content_type, "text/plain");
+	end_headers();
+
+	safe_write(1, header, strlen(header));
+	safe_write(1, msg, strlen(msg));
+	safe_write(1, "\n", 1);
+
+	exit(0);
+}
+
+static void not_found(void)
+{
+	send_status(404, "Not Found");
+	end_headers();
+}
+
+static void server_error(void)
+{
+	send_status(500, "Internal Error");
+	end_headers();
+}
+
+static void require_content_type(const char *need_type)
+{
+	const char *input_type = getenv("CONTENT_TYPE");
+	if (!input_type || strcmp(input_type, need_type))
+		invalid_request("Unsupported content-type");
+}
+
+static void do_GET_any_file(char *name)
+{
+	const char *p = git_path("%s", name);
+	struct stat sb;
+	uintmax_t remaining;
+	size_t n;
+	int fd = open(p, O_RDONLY);
+
+	if (fd < 0) {
+		not_found();
+		return;
+	}
+	if (fstat(fd, &sb) < 0) {
+		close(fd);
+		server_error();
+		die("fstat on plain file failed");
+	}
+	remaining = (uintmax_t)sb.st_size;
+
+	n = snprintf(buffer, sizeof(buffer),
+		"Content-Length: %" PRIuMAX "\r\n", remaining);
+	if (n >= sizeof(buffer))
+		die("protocol error: impossibly long header");
+	safe_write(1, buffer, n);
+	send_header(content_type, "application/octet-stream");
+	end_headers();
+
+	while (remaining) {
+		n = xread(fd, buffer, sizeof(buffer));
+		if (n < 0)
+			die("error reading from %s", p);
+		n = safe_write(1, buffer, n);
+		if (n <= 0)
+			break;
+	}
+	close(fd);
+}
+
+static int show_one_ref(const char *name, const unsigned char *sha1,
+	int flag, void *cb_data)
+{
+	struct object *o = parse_object(sha1);
+	if (!o)
+		return 0;
+
+	chunked_write("%s\t%s\n", sha1_to_hex(sha1), name);
+	if (o->type == OBJ_TAG) {
+		o = deref_tag(o, name, 0);
+		if (!o)
+			return 0;
+		chunked_write("%s\t%s^{}\n", sha1_to_hex(o->sha1), name);
+	}
+
+	return 0;
+}
+
+static void do_GET_info_refs(char *arg)
+{
+	send_header(content_type, "application/x-git-refs");
+	send_nocaching();
+	enable_chunking();
+	end_headers();
+
+	for_each_ref(show_one_ref, NULL);
+	end_chunking();
+}
+
+static void do_GET_info_packs(char *arg)
+{
+	size_t objdirlen = strlen(get_object_directory());
+	struct packed_git *p;
+
+	send_nocaching();
+	enable_chunking();
+	end_headers();
+
+	prepare_packed_git();
+	for (p = packed_git; p; p = p->next) {
+		if (!p->pack_local)
+			continue;
+		chunked_write("P %s\n", p->pack_name + objdirlen + 6);
+	}
+	chunked_write("\n");
+	end_chunking();
+}
+
+static void do_POST_receive_pack(char *arg)
+{
+	require_content_type("application/x-git-receive-pack");
+	send_header(content_type, "application/x-git-receive-pack-status");
+	send_nocaching();
+	send_connection_close();
+	end_headers();
+
+	execl_git_cmd("receive-pack",
+		"--report-status",
+		"--no-advertise-heads",
+		".",
+		NULL);
+	die("Failed to start receive-pack");
+}
+
+static struct service_cmd {
+	const char *method;
+	const char *pattern;
+	void (*imp)(char *);
+} services[] = {
+	{"GET", "/info/refs$", do_GET_info_refs},
+	{"GET", "/objects/info/packs", do_GET_info_packs},
+
+	{"GET", "/HEAD$", do_GET_any_file},
+	{"GET", "/objects/../.{38}$", do_GET_any_file},
+	{"GET", "/objects/pack/pack-[^/]*$", do_GET_any_file},
+	{"GET", "/objects/info/[^/]*$", do_GET_any_file},
+
+	{"POST", "/receive-pack", do_POST_receive_pack}
+};
+
+int main(int argc, char **argv)
+{
+	char *input_method = getenv("REQUEST_METHOD");
+	char *dir = getenv("PATH_TRANSLATED");
+	struct service_cmd *cmd = NULL;
+	char *cmd_arg = NULL;
+	int i;
+
+	if (!input_method)
+		die("No REQUEST_METHOD from server");
+	if (!strcmp(input_method, "HEAD"))
+		input_method = "GET";
+
+	if (!dir)
+		die("No PATH_TRANSLATED from server");
+
+	for (i = 0; i < ARRAY_SIZE(services); i++) {
+		struct service_cmd *c = &services[i];
+		regex_t re;
+		regmatch_t out[1];
+
+		if (strcmp(input_method, c->method))
+			continue;
+		if (regcomp(&re, c->pattern, REG_EXTENDED))
+			die("Bogus re in service table: %s", c->pattern);
+		if (!regexec(&re, dir, 2, out, 0)) {
+			size_t n = out[0].rm_eo - out[0].rm_so;
+			cmd = c;
+			cmd_arg = xmalloc(n);
+			strncpy(cmd_arg, dir + out[0].rm_so + 1, n);
+			cmd_arg[n] = 0;
+			dir[out[0].rm_so] = 0;
+			break;
+		}
+		regfree(&re);
+	}
+
+	if (!cmd)
+		invalid_request("Unsupported query request");
+
+	setup_path();
+	if (!enter_repo(dir, 0))
+		invalid_request("Not a Git repository");
+
+	cmd->imp(cmd_arg);
+	return 0;
+}
-- 
1.6.0.rc1.221.g9ae23

Re: [RFC 2/2] Add Git-aware CGI for Git-aware smart HTTP transport

[Junio C Hamano]

I very much like it.

But could you be a bit more explicit than application/x-git-refs magic?  I
suspect very strongly that clueless server operators would advertise the
type on repositories statically hosted there, and would defeat the point
of your patch.

We are not changing update-server-info so if we can find a place we can
use to hide the "magic", it would be a much more robust.

Perhaps "#" comment line in info/refs that is ignored on the reading side
but update-server-info never generates on its own?

Or perhaps sort the output differently from how update-server-info
produces its output, so that older client would not care but the magic
aware client can notice?

Re: [RFC 2/2] Add Git-aware CGI for Git-aware smart HTTP transport

[Shawn O. Pearce]

Junio C Hamano <gitster@pobox.com> wrote:
> But could you be a bit more explicit than application/x-git-refs magic?  I
> suspect very strongly that clueless server operators would advertise the
> type on repositories statically hosted there, and would defeat the point
> of your patch.

This is a very valid concern.  I started to worry about it myself
last night, but decided it was late enough and just wanted to start
the discussion on the list, extending JH's thread even further.
 
> Perhaps "#" comment line in info/refs that is ignored on the reading side
> but update-server-info never generates on its own?

This is a good idea.  I think anyone who consumes info/refs does
so with the understanding that "#" comment lines exist, and should
be skipped, but this is not something that has been heavily tested
in the wild yet.

My concern here goes back to the remark you made above. What if a
server owner mirrors a smart server by a non-Git aware device like
wget?  They will now have a copy of the info/refs content which will
suggest we have Git smarts on the backend, but really it isn't there.

Perhaps the smart server detection is something like:

	Smart Server Detection
	----------------------

	To detect a smart (Git-aware) server a client sends an
	empty POST request to info/refs; if a 200 OK response is
	received with the proper content type then the server can
	be assumed to be Git-aware, and the result contains the
	current info/refs data for that repository.

		C: POST /repository.git/info/refs HTTP/1.0
		C: Content-Length: 0

		S: HTTP/1.0 200 OK
		S: Content-Type: application/x-git-refs
		S:
		S: 95dcfa3633004da0049d3d0fa03f80589cbcaf31	refs/heads/maint

Then clients should just attempt this POST first before issuing
a GET info/refs.  Non Git-aware servers will issue an error code,
and the client can retry with a standard GET request, and assume
the server isn't a newer style.

-- 
Shawn.

Re: [RFC 2/2] Add Git-aware CGI for Git-aware smart HTTP transport

[Rogan Dawes]

Shawn O. Pearce wrote:

> Perhaps the smart server detection is something like:
> 
> 	Smart Server Detection
> 	----------------------
> 
> 	To detect a smart (Git-aware) server a client sends an
> 	empty POST request to info/refs; if a 200 OK response is
> 	received with the proper content type then the server can
> 	be assumed to be Git-aware, and the result contains the
> 	current info/refs data for that repository.
> 
> 		C: POST /repository.git/info/refs HTTP/1.0
> 		C: Content-Length: 0
> 
> 		S: HTTP/1.0 200 OK
> 		S: Content-Type: application/x-git-refs
> 		S:
> 		S: 95dcfa3633004da0049d3d0fa03f80589cbcaf31	refs/heads/maint
> 
> Then clients should just attempt this POST first before issuing
> a GET info/refs.  Non Git-aware servers will issue an error code,
> and the client can retry with a standard GET request, and assume
> the server isn't a newer style.
> 

I don't understand why you would want to keep the commands in the URL 
when you are doing a POST?

How about something like:

	C: POST /repository.git/ HTTP/1.0
	C: Content-Length: <calculated>
         C:
         C: <whatever command you want>

A dumb server will respond with:

	S: HTTP/1.1 405 Method not allowed

(expected according to the RFC)

Or

	S: HTTP/1.1 404 Not Found

(resulting from testing against my own repo :-) )

While a smart server will respond with a "200 Ok" and the results of the 
command.

Also, if everything is done via POST, you don't have to worry about a 
wget-cloned server appearing to be "smart", since no "smarts" will ever 
be returned in response to a GET request (and to the best of my 
knowledge, wget can't mirror using POST).

Rogan

Re: [RFC 2/2] Add Git-aware CGI for Git-aware smart HTTP transport

[Shawn O. Pearce]

Rogan Dawes <lists@dawes.za.net> wrote:
> Shawn O. Pearce wrote:
>>
>> 	Smart Server Detection
>> 	----------------------
>>
>> 	To detect a smart (Git-aware) server a client sends an
>> 	empty POST request to info/refs; [...]
>>
>> 		C: POST /repository.git/info/refs HTTP/1.0
>> 		C: Content-Length: 0
>
> I don't understand why you would want to keep the commands in the URL  
> when you are doing a POST?

Well, as Dscho pointed out this partly has to do with caching and
the transparent dumb server functionality.  By using the command in
the URL, and having the command match that of the dumb server file,
its easier to emulate a dumb server and also to permit caching.

Currently git-http-backend requests no caching for info/refs, but
I could see us tweaking that to permit several minutes of caching,
especially on big public sites like kernel.org.  Having info/refs
report stale by 5 minutes is not an issue when writes to there
already have a lag due to the master-slave mirroring system in use.

Because git-http-backend emulates a dumb server there is a command
dispatch table based upon the URL submitted.  Thus we already have
the command dispatch behavior implemented in the URL and doing it
in the POST body would only complicate the code further.

> Also, if everything is done via POST, you don't have to worry about a  
> wget-cloned server appearing to be "smart", since no "smarts" will ever  
> be returned in response to a GET request (and to the best of my  
> knowledge, wget can't mirror using POST).

I think we fixed the wget-cloned server issue by requesting
that clients use POST /info/refs to identify a smart server.
A wget-cloned repository will fail on this, and the client can
fallback to GET /info/refs and assume it must use the object
walker to fetch (or WebDAV to push).  A smart server would
respond to the POST /info/refs request correctly and the
client would know its smart.

-- 
Shawn.

Re: [RFC 2/2] Add Git-aware CGI for Git-aware smart HTTP transport

[H. Peter Anvin]

Shawn O. Pearce wrote:
> 
> Currently git-http-backend requests no caching for info/refs, but
> I could see us tweaking that to permit several minutes of caching,
> especially on big public sites like kernel.org.  Having info/refs
> report stale by 5 minutes is not an issue when writes to there
> already have a lag due to the master-slave mirroring system in use.
> 
> Because git-http-backend emulates a dumb server there is a command
> dispatch table based upon the URL submitted.  Thus we already have
> the command dispatch behavior implemented in the URL and doing it
> in the POST body would only complicate the code further.
> 

Let's put it this way: we're not seeing a huge amount of load from git 
protocol requests, and I'm going to assume "git+http" protocol to be 
used only by sites behind braindamaged firewalls (everyone else would 
use git protocol), so I'm not really all that worried about it.

I'm not sure if "emulating a dumb server" is desirable at all; it seems 
like it would at least in part defeat the purpose of minimizing the 
transaction count and otherwise be as much of a "smart" server as the 
medium permits.

	-hpa

Re: [RFC 2/2] Add Git-aware CGI for Git-aware smart HTTP transport

[Shawn O. Pearce]

"H. Peter Anvin" <hpa@zytor.com> wrote:
> Shawn O. Pearce wrote:
>>
>> Currently git-http-backend requests no caching for info/refs [...]
>
> Let's put it this way: we're not seeing a huge amount of load from git  
> protocol requests, and I'm going to assume "git+http" protocol to be  
> used only by sites behind braindamaged firewalls (everyone else would  
> use git protocol), so I'm not really all that worried about it.

Agreed.  There's another application I want git+http for, but that
may never materialize.  Or maybe it will someday.  I just have to
adopt a wait and see approach there.

> I'm not sure if "emulating a dumb server" is desirable at all; it seems  
> like it would at least in part defeat the purpose of minimizing the  
> transaction count and otherwise be as much of a "smart" server as the  
> medium permits.

I think it is a really good idea.  Then clients don't have to worry
about which HTTP URL is the "correct" one for them to be using.
End users will just magically get the smart git+http variant if
both sides support it and they need to use HTTP due to firewalls.
Clients will fall back onto the dumb protocol if the server doesn't
support smart clones.  Older clients (pre git+http) will still be
able to talk to a smart server, just slower.  This is nice for the
end user.  No thinking is required.

Never ask a human to do what a machine can do in less time.

I think its just 1 extra HTTP hit per fetch/push done against
a dumb server.  On a smart server that first hit will also give
us what we need to begin the conversation (the info/refs data).
On a dumb server its a wasted hit, but a dumb server is already
doing to suck.  One extra HTTP request against a dumb server is a
drop in the bucket.  Its also a pretty small request (an empty POST).

-- 
Shawn.

Re: [RFC 2/2] Add Git-aware CGI for Git-aware smart HTTP transport

[H. Peter Anvin]

Shawn O. Pearce wrote:
> 
>> I'm not sure if "emulating a dumb server" is desirable at all; it seems  
>> like it would at least in part defeat the purpose of minimizing the  
>> transaction count and otherwise be as much of a "smart" server as the  
>> medium permits.
> 
> I think it is a really good idea.  Then clients don't have to worry
> about which HTTP URL is the "correct" one for them to be using.
> End users will just magically get the smart git+http variant if
> both sides support it and they need to use HTTP due to firewalls.
> Clients will fall back onto the dumb protocol if the server doesn't
> support smart clones.  Older clients (pre git+http) will still be
> able to talk to a smart server, just slower.  This is nice for the
> end user.  No thinking is required.
> 
> Never ask a human to do what a machine can do in less time.
> 
> I think its just 1 extra HTTP hit per fetch/push done against
> a dumb server.  On a smart server that first hit will also give
> us what we need to begin the conversation (the info/refs data).
> On a dumb server its a wasted hit, but a dumb server is already
> doing to suck.  One extra HTTP request against a dumb server is a
> drop in the bucket.  Its also a pretty small request (an empty POST).
> 

Not arguing that URL compatibility isn't a good thing, but there are 
other ways to accomplish it, too.  After detecting either a smart or 
dumb server, we can use a redirect to point them to a different URL, as 
appropriate.

Furthermore, in the case of round-robin sites like kernel.org, this is 
actually *mandatory* in the case of a stateful server (we need a 
redirect to a server-specific URL), and highly recommended in the case 
of a stateless server (because of potential skew.)

	-hpa

Re: [RFC 2/2] Add Git-aware CGI for Git-aware smart HTTP transport

[Shawn O. Pearce]

"H. Peter Anvin" <hpa@zytor.com> wrote:
> Shawn O. Pearce wrote:
>>
>> I think it is a really good idea.  Then clients don't have to worry
>> about which HTTP URL is the "correct" one for them to be using.
>
> Not arguing that URL compatibility isn't a good thing, but there are  
> other ways to accomplish it, too.  After detecting either a smart or  
> dumb server, we can use a redirect to point them to a different URL, as  
> appropriate.

I'm not sure this is necessary.

Of course it all comes down to "how does an admin map Git repositories
into the URL space of the server"?

I thought it would be simple if the admin was able to map
repositories using a ScriptAlias and allow the server to perform
path info translation to give us the filesystem location of the
repository.  Then we don't have to configure our own map of the
available Git repositories.

Once you do that though you now have the URL space associated with
that repository served by a CGI.  For older clients we need to
either serve them the file, or issue a redirect to serve the file.
The redirect is messy because we need some configuration to explain
where the files are available in the server's URL space.

Or you go the other way, and have newer git+http clients try to
find the git aware server by a redirect.  Again we have to explain
where that git aware server is in the URL space of the server.

*sigh*

> Furthermore, in the case of round-robin sites like kernel.org, this is  
> actually *mandatory* in the case of a stateful server (we need a  
> redirect to a server-specific URL), and highly recommended in the case  
> of a stateless server (because of potential skew.)

Well, the git+http protocol will hold all state in the client, making
each RPC a stateless RPC operation.  The only issue is then dealing with
skew in a server farm.

I guess we need to ask client implementations to honor a redirect
on the first request and reuse that new base URL for all subsequent
requests that are part of the same "operation".  Then server farms
can issue a redirect to a server-specific hostname if a client
comes in with a round-robin DNS hostname, thus ensuring that for
this current operation there isn't skew.

-- 
Shawn.

Re: [RFC 2/2] Add Git-aware CGI for Git-aware smart HTTP transport

[H. Peter Anvin]

Shawn O. Pearce wrote:
> 
> I guess we need to ask client implementations to honor a redirect
> on the first request and reuse that new base URL for all subsequent
> requests that are part of the same "operation".  Then server farms
> can issue a redirect to a server-specific hostname if a client
> comes in with a round-robin DNS hostname, thus ensuring that for
> this current operation there isn't skew.
> 

Either that, or you can pass a "chase URL" in the payload of the 
request... it's more or less the same concept.

	-hpa

Add Git-aware CGI for Git-aware smart HTTP transport

[H. Peter Anvin]

Anything we can do to keep this moving forward?  I was extremely 
encouraged with the fast progress on this; this would be great to get to 
the point where we (kernel.org) can deploy it at least for testing.

	-hpa

Re: Add Git-aware CGI for Git-aware smart HTTP transport

[Shawn O. Pearce]

"H. Peter Anvin" <hpa@zytor.com> wrote:
> Anything we can do to keep this moving forward?  I was extremely  
> encouraged with the fast progress on this; this would be great to get to  
> the point where we (kernel.org) can deploy it at least for testing.

Sorry, I dropped it with my egit work.  I'll pick it up again and
try to continue it further.  I left off trying to implement the
push client and saying "damn, jgit is better structured to make this
sort of change than C git" and decided it was too late at night to
continue it more.  That was like a week ago.

-- 
Shawn.