[PATCH] RFC: proxy-command support for git://

[PATCH] RFC: proxy-command support for git://

[Paul Collins]

I spend some of my time using a network that only allows outgoing TCP
connections to certain ports, and the git-daemon port is not one of them.
This patch below implements an analogue to ssh's ProxyCommand feature
for git, as a less messy alternative to ssh port forwarding.  One can
use it to ssh to a bastion host and netcat to the destination:

  $ cat ~/bin/my-git-proxy-command
  #!/bin/sh
  exec ssh bastionhost nc "$1" "$2"

I've done a few pulls and a clone with it, and it seems to work.


Questions:

* Can git already do this and I just failed to notice?

* Where should git_use_proxy() look?  Some git configuration file?
  An environment variable?  Both?  Somewhere else?

It also needs to support non-default ports and probably other things I missed.


diff --git a/connect.c b/connect.c
index c2badc7..646e26f 100644
--- a/connect.c
+++ b/connect.c
@@ -448,6 +448,40 @@ static int git_tcp_connect(int fd[2], co
 
 #endif /* NO_IPV6 */
 
+static int git_proxy_connect(int fd[2], const char *prog, char *host, char *path)
+{
+	char *command = "my-git-proxy-command"; /* FIXME: cf. git_use_proxy() */
+	char *port = STR(DEFAULT_GIT_PORT);
+	int pipefd[2][2];
+	pid_t pid;
+
+	if (pipe(pipefd[0]) < 0 || pipe(pipefd[1]) < 0)
+		die("unable to create pipe pair for communication");
+	pid = fork();
+	if (!pid) {
+		dup2(pipefd[1][0], 0);
+		dup2(pipefd[0][1], 1);
+		close(pipefd[0][0]);
+		close(pipefd[0][1]);
+		close(pipefd[1][0]);
+		close(pipefd[1][1]);
+		execlp(command, command, host, port, NULL);
+		die("exec failed");
+	}
+	fd[0] = pipefd[0][0];
+	fd[1] = pipefd[1][1];
+	close(pipefd[0][1]);
+	close(pipefd[1][0]);
+	packet_write(fd[1], "%s %s\n", prog, path);
+	return pid;
+}
+
+static int git_use_proxy(void)
+{
+	/* FIXME: look for the proxy command somewhere - repo's config? environment? */
+	return 1;
+}
+
 /*
  * Yeah, yeah, fixme. Need to pass in the heads etc.
  */
@@ -482,8 +516,11 @@ int git_connect(int fd[2], char *url, co
 		}
 	}
 
-	if (protocol == PROTO_GIT)
+	if (protocol == PROTO_GIT) {
+		if (git_use_proxy())
+			return git_proxy_connect(fd, prog, host, path);
 		return git_tcp_connect(fd, prog, host, path);
+	}
 
 	if (pipe(pipefd[0]) < 0 || pipe(pipefd[1]) < 0)
 		die("unable to create pipe pair for communication");

-- 
Dag vijandelijk luchtschip de huismeester is dood

Re: [PATCH] RFC: proxy-command support for git://

[Junio C Hamano]

Paul Collins <paul@briny.ondioline.org> writes:

> I spend some of my time using a network that only allows outgoing TCP
> connections to certain ports, and the git-daemon port is not one of them.
> This patch below implements an analogue to ssh's ProxyCommand feature
> for git, as a less messy alternative to ssh port forwarding.

Wonderful.

> Questions:
>
> * Can git already do this and I just failed to notice?

Maybe I just failed to notice this too, but I do not think so.

> * Where should git_use_proxy() look?  Some git configuration file?
>   An environment variable?  Both?  Somewhere else?

My preference is put something in .git/config to describe which
proxy command (maybe the same one with different argument) to
use depending on where you are going.  When you have internal
hosts and external hosts you would want this to apply only to
external hosts.  Maybe you have two or more gateways and
depending on which external host you are going you may want to
use different proxied connection.  On top of the config file,
making it overridable from an environment variable would be
sensible.

Re: [PATCH] RFC: proxy-command support for git://

[Linus Torvalds]

On Thu, 3 Nov 2005, Junio C Hamano wrote:

> Paul Collins <paul@briny.ondioline.org> writes:
> 
> > I spend some of my time using a network that only allows outgoing TCP
> > connections to certain ports, and the git-daemon port is not one of them.
> > This patch below implements an analogue to ssh's ProxyCommand feature
> > for git, as a less messy alternative to ssh port forwarding.
> 
> Wonderful.
> 
> > Questions:
> >
> > * Can git already do this and I just failed to notice?
> 
> Maybe I just failed to notice this too, but I do not think so.

Actually, you could. TWO ways, in fact, afaik.

Just use the "ssh://host/pathname" format (or just "host:pathname") and 
the GIT_SSH environment variable.

You could also override the local command-name with

	git-send-pack --exec=my-local-send-program /machine/repo/path

where the "my-local-send-program" will parse /machine/repo/path thing. At 
least that works with git-send-pack, but it's possible it doesn't work 
with some other logic (ie "git push" might decide that it's unhappy that 
/machine/repo/path doesn't exist locally because it thinks it's a local 
path).

		Linus

Re: [PATCH] RFC: proxy-command support for git://

[Carl Baldwin]

Another way to do this would be using the ~/.ssh/config file.  It would
look something like this:

Host host
  ProxyCommand ...

Or, more generically...

Host *.*
  ProxyCommand ...

Host *.* assumes that if the machine name has a '.' in it then you are
trying to get outside the firewall.  This might not be a good assumption
but it works well where I am.

Then use host:pathname or ssh://host/pathname or whatever.

The advantage of using this is that it works for anything that uses ssh
to get outside the firewall.  Not just git.  So, setup is minimal.

Carl

On Thu, Nov 03, 2005 at 11:22:35AM -0800, Linus Torvalds wrote:
> 
> 
> On Thu, 3 Nov 2005, Junio C Hamano wrote:
> 
> > Paul Collins <paul@briny.ondioline.org> writes:
> > 
> > > I spend some of my time using a network that only allows outgoing TCP
> > > connections to certain ports, and the git-daemon port is not one of them.
> > > This patch below implements an analogue to ssh's ProxyCommand feature
> > > for git, as a less messy alternative to ssh port forwarding.
> > 
> > Wonderful.
> > 
> > > Questions:
> > >
> > > * Can git already do this and I just failed to notice?
> > 
> > Maybe I just failed to notice this too, but I do not think so.
> 
> Actually, you could. TWO ways, in fact, afaik.
> 
> Just use the "ssh://host/pathname" format (or just "host:pathname") and 
> the GIT_SSH environment variable.
> 
> You could also override the local command-name with
> 
> 	git-send-pack --exec=my-local-send-program /machine/repo/path
> 
> where the "my-local-send-program" will parse /machine/repo/path thing. At 
> least that works with git-send-pack, but it's possible it doesn't work 
> with some other logic (ie "git push" might decide that it's unhappy that 
> /machine/repo/path doesn't exist locally because it thinks it's a local 
> path).
> 
> 		Linus
> -
> To unsubscribe from this list: send the line "unsubscribe git" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

-- 
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
 Carl Baldwin                        Systems VLSI Laboratory
 Hewlett Packard Company
 MS 88                               work: 970 898-1523
 3404 E. Harmony Rd.                 work: Carl.N.Baldwin@hp.com
 Fort Collins, CO 80525              home: Carl@ecBaldwin.net
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Re: [PATCH] RFC: proxy-command support for git://

[Junio C Hamano]

Carl Baldwin <cnb@fc.hp.com> writes:

> Another way to do this would be using the ~/.ssh/config file.  It would
> look something like this:

Maybe I am slow today, but wouldn't this require the other end
(i.e. remote repo) to let you ssh in?

I think the point of the original patch was to give proxied
connection to git:// transport, not git over ssh.

Re: [PATCH] v2: proxy-command support for git://

[Paul Collins]

Junio C Hamano <junkio@cox.net> writes:

> Paul Collins <paul@briny.ondioline.org> writes:
>
>> * Where should git_use_proxy() look?  Some git configuration file?
>>   An environment variable?  Both?  Somewhere else?
>
> My preference is put something in .git/config to describe which
> proxy command (maybe the same one with different argument) to
> use depending on where you are going.  When you have internal
> hosts and external hosts you would want this to apply only to
> external hosts.  Maybe you have two or more gateways and
> depending on which external host you are going you may want to
> use different proxied connection.  On top of the config file,
> making it overridable from an environment variable would be
> sensible.

Here is an updated patch that first looks for GIT_PROXY_COMMAND in the
environment and then git.proxycommand in the repository's
configuration file.  I have left the calling convention the same --
argv[1] is the host and argv[2] is the port.

I've taken the hostname parsing verbatim from git_tcp_connect(), so it
should now support an explicit port number and whatever that business
with the square brackets is.  (Should I move this to a helper function?)

Regarding internal vs. external hosts, the proxy command can simply
run netcat locally to internal hosts, so perhaps that is sufficient.


diff --git a/connect.c b/connect.c
index c2badc7..43eec67 100644
--- a/connect.c
+++ b/connect.c
@@ -448,6 +448,73 @@ static int git_tcp_connect(int fd[2], co
 
 #endif /* NO_IPV6 */
 
+static char *git_proxy_command = NULL;
+
+static int git_proxy_command_options(const char *var, const char *value)
+{
+	if (git_proxy_command == NULL) {
+		if (!strcmp(var, "git.proxycommand")) {
+			git_proxy_command = xmalloc(strlen(value) + 1);
+			strcpy(git_proxy_command, value);
+			return 0;
+		}
+	}
+
+	return git_default_config(var, value);
+}
+
+static int git_use_proxy(void)
+{
+	git_proxy_command = getenv("GIT_PROXY_COMMAND");
+	git_config(git_proxy_command_options);
+	return git_proxy_command != NULL;
+}
+
+static int git_proxy_connect(int fd[2], const char *prog, char *host, char *path)
+{
+	char *port = STR(DEFAULT_GIT_PORT);
+	char *colon, *end;
+	int pipefd[2][2];
+	pid_t pid;
+
+	if (host[0] == '[') {
+		end = strchr(host + 1, ']');
+		if (end) {
+			*end = 0;
+			end++;
+			host++;
+		} else
+			end = host;
+	} else
+		end = host;
+	colon = strchr(end, ':');
+
+	if (colon) {
+		*colon = 0;
+		port = colon + 1;
+	}
+
+	if (pipe(pipefd[0]) < 0 || pipe(pipefd[1]) < 0)
+		die("unable to create pipe pair for communication");
+	pid = fork();
+	if (!pid) {
+		dup2(pipefd[1][0], 0);
+		dup2(pipefd[0][1], 1);
+		close(pipefd[0][0]);
+		close(pipefd[0][1]);
+		close(pipefd[1][0]);
+		close(pipefd[1][1]);
+		execlp(git_proxy_command, git_proxy_command, host, port, NULL);
+		die("exec failed");
+	}
+	fd[0] = pipefd[0][0];
+	fd[1] = pipefd[1][1];
+	close(pipefd[0][1]);
+	close(pipefd[1][0]);
+	packet_write(fd[1], "%s %s\n", prog, path);
+	return pid;
+}
+
 /*
  * Yeah, yeah, fixme. Need to pass in the heads etc.
  */
@@ -482,8 +549,11 @@ int git_connect(int fd[2], char *url, co
 		}
 	}
 
-	if (protocol == PROTO_GIT)
+	if (protocol == PROTO_GIT) {
+		if (git_use_proxy())
+			return git_proxy_connect(fd, prog, host, path);
 		return git_tcp_connect(fd, prog, host, path);
+	}
 
 	if (pipe(pipefd[0]) < 0 || pipe(pipefd[1]) < 0)
 		die("unable to create pipe pair for communication");


-- 
Dag vijandelijk luchtschip de huismeester is dood

Re: [PATCH] v2: proxy-command support for git://

[Junio C Hamano]

Paul Collins <paul@briny.ondioline.org> writes:

> Regarding internal vs. external hosts, the proxy command can simply
> run netcat locally to internal hosts, so perhaps that is sufficient.

I was hoping this to become a bit more generalized mechanism
than that; for example using outgoing plug over HTTP Connect or
telnet proxy using tn-gw-nav.

Re: [PATCH] v2: proxy-command support for git://

[Junio C Hamano]

Junio C Hamano <junkio@cox.net> writes:

> Paul Collins <paul@briny.ondioline.org> writes:
>
>> Regarding internal vs. external hosts, the proxy command can simply
>> run netcat locally to internal hosts, so perhaps that is sufficient.
>
> I was hoping this to become a bit more generalized mechanism
> than that; for example using outgoing plug over HTTP Connect or
> telnet proxy using tn-gw-nav.

I realize the above does not really convey my real objection.

Your "ssh to the proxy/firewall host and run netcat to the
destination" would not work for me to reach the internal hosts
at all (while it would work for external ones), because my
firewall does not know names of our internal hosts (the same for
using tn-gw-nav to cross http or telnet proxy).

Re: [PATCH] v2: proxy-command support for git://

[Paul Collins]

Junio C Hamano <junkio@cox.net> writes:

> Junio C Hamano <junkio@cox.net> writes:
>
>> Paul Collins <paul@briny.ondioline.org> writes:
>>
>>> Regarding internal vs. external hosts, the proxy command can simply
>>> run netcat locally to internal hosts, so perhaps that is sufficient.
>>
>> I was hoping this to become a bit more generalized mechanism
>> than that; for example using outgoing plug over HTTP Connect or
>> telnet proxy using tn-gw-nav.

"Run a program and talk to it via stdin/stdout" is as general as it
gets, isn't it?  ssh+netcat is just what I happen to use.

> I realize the above does not really convey my real objection.
>
> Your "ssh to the proxy/firewall host and run netcat to the
> destination" would not work for me to reach the internal hosts
> at all (while it would work for external ones), because my
> firewall does not know names of our internal hosts (the same for
> using tn-gw-nav to cross http or telnet proxy).

It doesn't have to be unconditional.  For example, one could have:

    if on_blargco_network; then
        # internal
        case "$1" in
            *.blargco.com)
                exec nc "$1" "$2"
                ;;
            *)
                exec ssh bastion.blargco.com nc "$1" "$2"
                ;;
        esac
    else
        # external
        case "$1" in
            *.blargco.com)
                exec ssh bastion.blargco.com nc "$1" "$2"
                ;;
            *)
                exec ssh bastion nc "$1" "$2"
                ;;
        esac
    fi

But perhaps I do not really understand your objection.

-- 
Dag vijandelijk luchtschip de huismeester is dood

Re: [PATCH] v2: proxy-command support for git://

[Junio C Hamano]

Paul Collins <paul@briny.ondioline.org> writes:

> But perhaps I do not really understand your objection.

No, I think you are getting it right.

I just wanted to avoid using the proxy script for some hosts,
depending on where you are going.  Obviously you can teach the
proxy script to do passthru for some hosts like you did in your
message.  The only difference is where the configuration is
specified.  I wanted it to be in the git configuration file
(i.e. using different proxy script or no script, depending on
the host).  Your example has that configuration wired in the
single script that is always called regardless of the
destination, and the script itself switches how it proxies,
depending on where it is going, perhaps using its own
configuration file or hardcoding.

Re: [PATCH] v2: proxy-command support for git://

[Paul Collins]

Junio C Hamano <junkio@cox.net> writes:

> Paul Collins <paul@briny.ondioline.org> writes:
>
>> But perhaps I do not really understand your objection.
>
> No, I think you are getting it right.
>
> I just wanted to avoid using the proxy script for some hosts,
> depending on where you are going.  Obviously you can teach the
> proxy script to do passthru for some hosts like you did in your
> message.  The only difference is where the configuration is
> specified.  I wanted it to be in the git configuration file
> (i.e. using different proxy script or no script, depending on
> the host).  Your example has that configuration wired in the
> single script that is always called regardless of the
> destination, and the script itself switches how it proxies,
> depending on where it is going, perhaps using its own
> configuration file or hardcoding.

I had some ideas along those lines, but I didn't like any of them.

 * Extend the proxy-command "protocol" with a third argument, an
   action.  For example if 'query $host $port' returns successfully,
   then git should run it with arguments 'connect $host $port',
   otherwise use git_tcp_connect().

 * Add a Proxy-Command field to the files in .git/remotes, e.g.:

     URL: git://git.kernel.org/pub/scm/git/git.git/
     Pull: master:origin
     Proxy-command: my-git-proxy-command

 * If the git config syntax is extended to allow dots in section or
   key names:

     [proxy]
     git.kernel.org = "ssh-to-bastion-proxy-command"
     git.blargco.com = "blargco-proxy-command"

   or perhaps

     [git.kernel.org]
     proxycommand = "ssh-to-bastion-proxy-command"
     [git.blargco.com]
     proxycommand = "blargco-proxy-command"

-- 
Dag vijandelijk luchtschip de huismeester is dood

Re: [PATCH] v2: proxy-command support for git://

[Linus Torvalds]

On Fri, 4 Nov 2005, Paul Collins wrote:
> 
>  * If the git config syntax is extended to allow dots in section or
>    key names:
> 
>      [proxy]
>      git.kernel.org = "ssh-to-bastion-proxy-command"
>      git.blargco.com = "blargco-proxy-command"

I would suggest

	[proxy]
		command = "ssh-to-bastion-proxy-command" for git.kernel.org

and then it's easy enough to just parse the value "proxy.command" with 
code like

	host = value;
	cmd = strstr(value, " for ");
	if (!cmd)
		return -1;
	*cmd = 0;
	cmd += 5;

which would do the right thing..

The thing is, it's not just ".". I could well imagine that you'd have

	[proxy]
		command="ssh" for "ssh://kernel.org/"
		command="proxy-command" for kernel.org
		command="myprotocol-command" for "my://"

which would actually allow you to literally add your own protocol names 
(it would see that the target starts with "my://", and decide that it 
shoul drun the "myprotocol-command" for the proxy).

I'd rather allow free-form strings for the values than for the key names. 
If we allow free-form key-names, then random text files suddenly often 
become valid (but strange) config files.

Right now the non-free-form key names are the strongest syntax checker of 
the whole protocol.

		Linus

[PATCH] git-proxy updates.

[Junio C Hamano]

Paul Collins <paul@briny.ondioline.org> writes:

> Here is an updated patch that first looks for GIT_PROXY_COMMAND in the
> environment and then git.proxycommand in the repository's
> configuration file.

Sorry for holding onto your patch without any action (other than
looking at it, which from your end you cannot tell I was doing
;-).  I think we will benefit from your patch in some form, and
here is what I tried on top to update it to the multivalue
configuration syntax Linus suggested during the list discussion.
Requesting for comments the original author and from the general
public.

I suspect this touches the same area as the user-path series
Andreas Ericsson has been working on, so it might need some
adjusting after that patch series goes in.  I'll see that soon
enough when I make the proposed updates branch tomorrow.

 -- >8 --

This builds on top of the git-proxy mechanism Paul Collins did,
and updates its configuration mechanism.

 * GIT_PROXY_COMMAND environment variable is used as the
   catch-all fallback, as in the original.  This has not
   changed.

 * Renames proxy configuration variables to core.gitproxy; this
   has become a multi-value variable per list discussion, most
   notably from suggestion by Linus.

	[core]
	;# matches www.kernel.org as well
	gitproxy = netcatter for kernel.org
	gitproxy = netscatter for sample.xz
	gitproxy = none for mydomain.xz
	gitproxy = netcatter-default

   The values are command names, followed by an optional " for "
   and domainname; the first tail-match of the domainname
   determines which proxy command is used.  An entry without "
   for " matches any domain and can be used as the default.

   The command name "none" is special -- it tells the mechanism
   not to use any proxy command and use the native git://
   connection.

Signed-off-by: Junio C Hamano <junkio@cox.net>

---

 connect.c |   52 +++++++++++++++++++++++++++++++++++++++++++++-------
 1 files changed, 45 insertions(+), 7 deletions(-)

applies-to: d4a5ace05fd892cb455f330d379cdaa69dc6005c
c1d7ac7b4680202d3549a72188f372367ac8837c
diff --git a/connect.c b/connect.c
index 43eec67..11a804f 100644
--- a/connect.c
+++ b/connect.c
@@ -449,25 +449,63 @@ static int git_tcp_connect(int fd[2], co
 #endif /* NO_IPV6 */
 
 static char *git_proxy_command = NULL;
+static const char *rhost_name = NULL;
+static int rhost_len;
 
 static int git_proxy_command_options(const char *var, const char *value)
 {
-	if (git_proxy_command == NULL) {
-		if (!strcmp(var, "git.proxycommand")) {
-			git_proxy_command = xmalloc(strlen(value) + 1);
-			strcpy(git_proxy_command, value);
+	if (!strcmp(var, "core.gitproxy")) {
+		if (git_proxy_command)
 			return 0;
+		/* [core]
+		 * ;# matches www.kernel.org as well
+		 * gitproxy = netcatter-1 for kernel.org
+		 * gitproxy = netcatter-2 for sample.xz
+		 * gitproxy = netcatter-default
+		 */
+		const char *for_pos = strstr(value, " for ");
+		int matchlen = -1;
+		int hostlen;
+
+		if (!for_pos)
+			/* matches everybody */
+			matchlen = strlen(value);
+		else {
+			hostlen = strlen(for_pos + 5);
+			if (rhost_len < hostlen)
+				matchlen = -1;
+			else if (!strncmp(for_pos + 5,
+					  rhost_name + rhost_len - hostlen,
+					  hostlen) &&
+				 ((rhost_len == hostlen) ||
+				  rhost_name[rhost_len - hostlen -1] == '.'))
+				matchlen = for_pos - value;
+			else
+				matchlen = -1;
+		}
+		if (0 <= matchlen) {
+			/* core.gitproxy = none for kernel.org */
+			if (matchlen == 4 && 
+			    !memcmp(value, "none", 4))
+				matchlen = 0;
+			git_proxy_command = xmalloc(matchlen + 1);
+			memcpy(git_proxy_command, value, matchlen);
+			git_proxy_command[matchlen] = 0;
 		}
+		return 0;
 	}
 
 	return git_default_config(var, value);
 }
 
-static int git_use_proxy(void)
+static int git_use_proxy(const char *host)
 {
+	rhost_name = host;
+	rhost_len = strlen(host);
 	git_proxy_command = getenv("GIT_PROXY_COMMAND");
 	git_config(git_proxy_command_options);
-	return git_proxy_command != NULL;
+	rhost_name = NULL;
+	return (git_proxy_command && *git_proxy_command);
 }
 
 static int git_proxy_connect(int fd[2], const char *prog, char *host, char *path)
@@ -550,7 +588,7 @@ int git_connect(int fd[2], char *url, co
 	}
 
 	if (protocol == PROTO_GIT) {
-		if (git_use_proxy())
+		if (git_use_proxy(host))
 			return git_proxy_connect(fd, prog, host, path);
 		return git_tcp_connect(fd, prog, host, path);
 	}
---
0.99.9.GIT