FW: git via http protocol _and_ a proxy using NTLM authentication -- git 1.5.4.2 & curl 7.18.0

FW: git via http protocol _and_ a proxy using NTLM authentication -- git 1.5.4.2 & curl 7.18.0

[Ken.Fuchs]

Need help to modify git to work with the http protocol _and_
a proxy using HTLM authentication.

--

The following changes were made to http.c (for NTLM authentication):

$ diff -c /mnt/nfs/git/git-1.5.4.2/http.c.orig \
    /mnt/nfs/git/git-1.5.4.2/http.c
--- 215,220 ----
--- 215,223 ----
        if (curl_http_proxy)
                curl_easy_setopt(result, CURLOPT_PROXY,
curl_http_proxy);

+       curl_easy_setopt(result, CURLOPT_PROXYAUTH,
(long)CURLAUTH_NTLM);
+       curl_easy_setopt(result, CURLOPT_PROXYUSERPWD,
+               "<user-id>:<password>");
+
        return result;
  }
$

--

git fetch _not_ working via an NTML proxy:

$ http_proxy=<proxy-domain>:<proxy-port> GIT_CURL_VERBOSE=1 \
    git fetch http://www.kernel.org/pub/scm/git/git.git

[A few paragraphs of proxy authentication debug output that
BogoFilter apparently gives a high SPAM quotient to, so I've
given up trying to include it.]

fatal: Couldn't find remote ref HEAD
$ 

It seems that git fetch (via HTLM proxy) works until

> fatal: Couldn't find remote ref HEAD

git 1.5.4.2 dies on line 158 of builtin-fetch.c,
because the call to get_remote_ref() in remote.c fails.

Does anything need to be changed in remote.c?

Any other suggestions?

Thanks,

Ken Fuchs

Re: FW: git via http protocol _and_ a proxy using NTLM authentication -- git 1.5.4.2 & curl 7.18.0

[Daniel Stenberg]

On Tue, 26 Feb 2008, Ken.Fuchs@bench.com wrote:

> +       curl_easy_setopt(result, CURLOPT_PROXYAUTH, (long)CURLAUTH_NTLM);
> +       curl_easy_setopt(result, CURLOPT_PROXYUSERPWD,
> +               "<user-id>:<password>");

First, you should rather allow any auth and not just the specific one you 
want.

Then, the userid and password is probably better passed in embedded in the 
proxy URL as that's given on the command line/environment already. Or as 
separate arguments.

> It seems that git fetch (via HTLM proxy) works until
>
>> fatal: Couldn't find remote ref HEAD

Well, the CURLOPT_PROXY is set in transport.c as well which your patch didn't 
address. If that's the case, I figure the verbose output should've shown some 
auth failures with the proxy?

RE: FW: git via http protocol _and_ a proxy using NTLM authentication -- git 1.5.4.2 & curl 7.18.0

[Ken.Fuchs]

> Ken Fuchs wrote:
> 
> > +       curl_easy_setopt(result, CURLOPT_PROXYAUTH, 
> (long)CURLAUTH_NTLM);
> > +       curl_easy_setopt(result, CURLOPT_PROXYUSERPWD,
> > +               "<user-id>:<password>");

Daniel Stenberg wrote:

> First, you should rather allow any auth and not just the 
> specific one you want.
> 
> Then, the userid and password is probably better passed in 
> embedded in the proxy URL as that's given on the command
> line/environment already. Or as separate arguments.

Agreed.  (I'd just like to get it working first.)

> > It seems that git fetch (via HTLM proxy) works until
> >
> >> fatal: Couldn't find remote ref HEAD
> 
> Well, the CURLOPT_PROXY is set in transport.c as well which 
> your patch didn't address.

Thanks, I just did a similar patch to transport.c:

$ diff -u ../git-1.5.4.2/transport.c.orig \
          ../git-1.5.4.2/transport.c
@@ -456,7 +456,8 @@
        if (transport->remote->http_proxy)
                curl_easy_setopt(slot->curl, CURLOPT_PROXY,
                                 transport->remote->http_proxy);
-
+       curl_easy_setopt(slot->curl, CURLOPT_PROXYAUTH,
(long)CURLAUTH_NTLM);
+       curl_easy_setopt(slot->curl, CURLOPT_PROXYUSERPWD,
"<user-id>:<password>");
        if (start_active_slot(slot)) {
                run_active_slot(slot);
                if (results.curl_result != CURLE_OK) {
$

> If that's the case, I figure the verbose output 
> should've shown some auth failures with the proxy?

No, actually the last 2 lines of debug output (unchanged) is:

* Connection #0 to host <proxy domain> left intact
fatal: Couldn't find remote ref HEAD

So, the proxy communication via NTLM authentication seems to be working.
The patch to transport.c did not change anything as far as I can see.

The fatal error is from remote.c.  Perhaps, it also requires some
changes.

--

Example of curl (sans git) working via an NTLM proxy:

$ curl --proxy-ntlm --proxy-user <user-id> \
    --proxy <proxy-domain>:<proxy-port> http://slashdot.org/
Enter proxy password for user '<user-id>': <non-echoed password>
<HTML of /. home page>

Thus, it seems that git could be modified to work via HTLM
authentication, but my simple changes to http.c and transport.c
above are probably not sufficient.

Suggestions are most welcome.

Thanks,

Ken Fuchs

Re: FW: git via http protocol _and_ a proxy using NTLM authentication -- git 1.5.4.2 & curl 7.18.0

[Mike Hommey]

On Tue, Feb 26, 2008 at 05:46:21PM -0600, Ken.Fuchs@bench.com wrote:
> > Well, the CURLOPT_PROXY is set in transport.c as well which 
> > your patch didn't address.
> 
> Thanks, I just did a similar patch to transport.c:
> 
> $ diff -u ../git-1.5.4.2/transport.c.orig \
>           ../git-1.5.4.2/transport.c
> @@ -456,7 +456,8 @@
>         if (transport->remote->http_proxy)
>                 curl_easy_setopt(slot->curl, CURLOPT_PROXY,
>                                  transport->remote->http_proxy);
> -
> +       curl_easy_setopt(slot->curl, CURLOPT_PROXYAUTH,
> (long)CURLAUTH_NTLM);
> +       curl_easy_setopt(slot->curl, CURLOPT_PROXYUSERPWD,
> "<user-id>:<password>");
>         if (start_active_slot(slot)) {
>                 run_active_slot(slot);
>                 if (results.curl_result != CURLE_OK) {
> $

Starting with curl 7.14.1, you're supposed to be able to use the
http://user:pass@proxy/ syntax, though I'm not sure it deals well with
NTLM domains. You can probably leave CURLOPT_PROXYUSERPWD out if you
set your proxy url correctly.

As for CURLOPT_PROXYAUTH, it would be better to set it from another
config.

Note that remote.<name>.proxy config doesn't work as expected, you
should use http.proxy which just work (and the change in transport.c is
useless, then). I have, as part of by http-refactoring topic, a patch
for remote.<name>.proxy to work better, though it doesn't support
changing the proxy authentication method.

> > If that's the case, I figure the verbose output 
> > should've shown some auth failures with the proxy?
> 
> No, actually the last 2 lines of debug output (unchanged) is:
> 
> * Connection #0 to host <proxy domain> left intact
> fatal: Couldn't find remote ref HEAD
> 
> So, the proxy communication via NTLM authentication seems to be working.
> The patch to transport.c did not change anything as far as I can see.
> 
> The fatal error is from remote.c.  Perhaps, it also requires some
> changes.

Does your remote have a HEAD ref ?

Mike

Re: FW: git via http protocol _and_ a proxy using NTLM authentication -- git 1.5.4.2 & curl 7.18.0

[Mike Hommey]

On Wed, Feb 27, 2008 at 08:20:12AM +0100, Mike Hommey wrote:
> On Tue, Feb 26, 2008 at 05:46:21PM -0600, Ken.Fuchs@bench.com wrote:
> > > Well, the CURLOPT_PROXY is set in transport.c as well which 
> > > your patch didn't address.
> > 
> > Thanks, I just did a similar patch to transport.c:
> > 
> > $ diff -u ../git-1.5.4.2/transport.c.orig \
> >           ../git-1.5.4.2/transport.c
> > @@ -456,7 +456,8 @@
> >         if (transport->remote->http_proxy)
> >                 curl_easy_setopt(slot->curl, CURLOPT_PROXY,
> >                                  transport->remote->http_proxy);
> > -
> > +       curl_easy_setopt(slot->curl, CURLOPT_PROXYAUTH,
> > (long)CURLAUTH_NTLM);
> > +       curl_easy_setopt(slot->curl, CURLOPT_PROXYUSERPWD,
> > "<user-id>:<password>");
> >         if (start_active_slot(slot)) {
> >                 run_active_slot(slot);
> >                 if (results.curl_result != CURLE_OK) {
> > $
> 
> Starting with curl 7.14.1, you're supposed to be able to use the
> http://user:pass@proxy/ syntax, though I'm not sure it deals well with
> NTLM domains. You can probably leave CURLOPT_PROXYUSERPWD out if you
> set your proxy url correctly.
> 
> As for CURLOPT_PROXYAUTH, it would be better to set it from another
> config.

Or we should set it to CURLOPT_AUTHANY by default.

Mike

[PATCH] Set proxy override with http_init()

[Mike Hommey]

In transport.c, proxy setting (the one from the remote conf) was set through
curl_easy_setopt() call, while http.c already does the same with the
http.proxy setting. We now just use this infrastructure instead, and make
http_init() now take the proxy url as argument.

At the same time, we make get_http_walker() take a proxy argument too, and
pass it to http_init(), which makes remote defined proxy be used for more
than get_refs_via_curl().

Signed-off-by: Mike Hommey <mh@glandium.org>
---
 > Note that remote.<name>.proxy config doesn't work as expected, you
 > should use http.proxy which just work (and the change in transport.c is
 > useless, then). I have, as part of by http-refactoring topic, a patch
 > for remote.<name>.proxy to work better, though it doesn't support
 > changing the proxy authentication method.
 
 And here is said patch.

 http-push.c   |    2 +-
 http-walker.c |    4 ++--
 http.c        |   10 +++++++++-
 http.h        |    2 +-
 transport.c   |    9 ++++-----
 walker.h      |    2 +-
 6 files changed, 18 insertions(+), 11 deletions(-)

diff --git a/http-push.c b/http-push.c
index 0beb740..04e056d 100644
--- a/http-push.c
+++ b/http-push.c
@@ -2240,7 +2240,7 @@ int main(int argc, char **argv)
 
 	memset(remote_dir_exists, -1, 256);
 
-	http_init();
+	http_init(NULL);
 
 	no_pragma_header = curl_slist_append(no_pragma_header, "Pragma:");
 
diff --git a/http-walker.c b/http-walker.c
index 2c37868..02be6c8 100644
--- a/http-walker.c
+++ b/http-walker.c
@@ -902,13 +902,13 @@ static void cleanup(struct walker *walker)
 	curl_slist_free_all(data->no_pragma_header);
 }
 
-struct walker *get_http_walker(const char *url)
+struct walker *get_http_walker(const char *url, const char *proxy)
 {
 	char *s;
 	struct walker_data *data = xmalloc(sizeof(struct walker_data));
 	struct walker *walker = xmalloc(sizeof(struct walker));
 
-	http_init();
+	http_init(proxy);
 
 	data->no_pragma_header = curl_slist_append(NULL, "Pragma:");
 
diff --git a/http.c b/http.c
index 519621a..89194d7 100644
--- a/http.c
+++ b/http.c
@@ -219,13 +219,16 @@ static CURL* get_curl_handle(void)
 	return result;
 }
 
-void http_init(void)
+void http_init(const char *proxy)
 {
 	char *low_speed_limit;
 	char *low_speed_time;
 
 	curl_global_init(CURL_GLOBAL_ALL);
 
+	if (proxy)
+		curl_http_proxy = xstrdup(proxy);
+
 	pragma_header = curl_slist_append(pragma_header, "Pragma: no-cache");
 
 #ifdef USE_CURL_MULTI
@@ -315,6 +318,11 @@ void http_cleanup(void)
 
 	curl_slist_free_all(pragma_header);
 	pragma_header = NULL;
+
+	if (curl_http_proxy) {
+		free(curl_http_proxy);
+		curl_http_proxy = NULL;
+	}
 }
 
 struct active_request_slot *get_active_slot(void)
diff --git a/http.h b/http.h
index 9bab2c8..dcd5cea 100644
--- a/http.h
+++ b/http.h
@@ -83,7 +83,7 @@ extern void add_fill_function(void *data, int (*fill)(void *));
 extern void step_active_slots(void);
 #endif
 
-extern void http_init(void);
+extern void http_init(const char *proxy);
 extern void http_cleanup(void);
 
 extern int data_received;
diff --git a/transport.c b/transport.c
index 397983d..26f0f02 100644
--- a/transport.c
+++ b/transport.c
@@ -442,7 +442,8 @@ static struct ref *get_refs_via_curl(struct transport *transport)
 	struct ref *last_ref = NULL;
 
 	if (!transport->data)
-		transport->data = get_http_walker(transport->url);
+		transport->data = get_http_walker(transport->url,
+						transport->remote->http_proxy);
 
 	refs_url = xmalloc(strlen(transport->url) + 11);
 	sprintf(refs_url, "%s/info/refs", transport->url);
@@ -453,9 +454,6 @@ static struct ref *get_refs_via_curl(struct transport *transport)
 	curl_easy_setopt(slot->curl, CURLOPT_WRITEFUNCTION, fwrite_buffer);
 	curl_easy_setopt(slot->curl, CURLOPT_URL, refs_url);
 	curl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, NULL);
-	if (transport->remote->http_proxy)
-		curl_easy_setopt(slot->curl, CURLOPT_PROXY,
-				 transport->remote->http_proxy);
 
 	if (start_active_slot(slot)) {
 		run_active_slot(slot);
@@ -509,7 +507,8 @@ static int fetch_objs_via_curl(struct transport *transport,
 				 int nr_objs, struct ref **to_fetch)
 {
 	if (!transport->data)
-		transport->data = get_http_walker(transport->url);
+		transport->data = get_http_walker(transport->url,
+						transport->remote->http_proxy);
 	return fetch_objs_via_walker(transport, nr_objs, to_fetch);
 }
 
diff --git a/walker.h b/walker.h
index ea2c363..2cc448a 100644
--- a/walker.h
+++ b/walker.h
@@ -32,6 +32,6 @@ int walker_fetch(struct walker *impl, int targets, char **target,
 
 void walker_free(struct walker *walker);
 
-struct walker *get_http_walker(const char *url);
+struct walker *get_http_walker(const char *url, const char *proxy);
 
 #endif /* WALKER_H */
-- 
1.5.4.1.48.g0d77

Re: [PATCH] Set proxy override with http_init()

[Junio C Hamano]

Mike Hommey <mh@glandium.org> writes:

> In transport.c, proxy setting (the one from the remote conf) was set through
> curl_easy_setopt() call, while http.c already does the same with the
> http.proxy setting. We now just use this infrastructure instead, and make
> http_init() now take the proxy url as argument.
>
> At the same time, we make get_http_walker() take a proxy argument too, and
> pass it to http_init(), which makes remote defined proxy be used for more
> than get_refs_via_curl().
>
> Signed-off-by: Mike Hommey <mh@glandium.org>
> ---

Thanks.  I am kicking this to Daniel for a review ;-).

Re: [PATCH] Set proxy override with http_init()

[Daniel Barkalow]

On Wed, 27 Feb 2008, Mike Hommey wrote:

> In transport.c, proxy setting (the one from the remote conf) was set through
> curl_easy_setopt() call, while http.c already does the same with the
> http.proxy setting. We now just use this infrastructure instead, and make
> http_init() now take the proxy url as argument.
> 
> At the same time, we make get_http_walker() take a proxy argument too, and
> pass it to http_init(), which makes remote defined proxy be used for more
> than get_refs_via_curl().

It's a good idea, but maybe http_init() (and the call chain leading up to 
it) should take the struct remote, so that it can get anything else you 
might also have there in the future? For that matter, is it intentional 
that http-push ignore the proxy setting, which is equally available in 
http-push, but not used? If so, it should probably get a comment.

	-Daniel
*This .sig left intentionally blank*

Re: [PATCH] Set proxy override with http_init()

[Mike Hommey]

On Wed, Feb 27, 2008 at 02:59:38PM -0500, Daniel Barkalow wrote:
> On Wed, 27 Feb 2008, Mike Hommey wrote:
> 
> > In transport.c, proxy setting (the one from the remote conf) was set through
> > curl_easy_setopt() call, while http.c already does the same with the
> > http.proxy setting. We now just use this infrastructure instead, and make
> > http_init() now take the proxy url as argument.
> > 
> > At the same time, we make get_http_walker() take a proxy argument too, and
> > pass it to http_init(), which makes remote defined proxy be used for more
> > than get_refs_via_curl().
> 
> It's a good idea, but maybe http_init() (and the call chain leading up to 
> it) should take the struct remote, so that it can get anything else you 
> might also have there in the future? For that matter, is it intentional 
> that http-push ignore the proxy setting, which is equally available in 
> http-push, but not used? If so, it should probably get a comment.

It is intentional, for the moment. I'll rework the patch to take the
struct remote, and add a comment.

Mike

[PATCH] Set proxy override with http_init()

[Mike Hommey]

In transport.c, proxy setting (the one from the remote conf) was set through
curl_easy_setopt() call, while http.c already does the same with the
http.proxy setting. We now just use this infrastructure instead, and make
http_init() now take the struct remote as argument so that it can take the
http_proxy setting from there, and any other property that would be added
later.

At the same time, we make get_http_walker() take a struct remote argument
too, and pass it to http_init(), which makes remote defined proxy be used
for more than get_refs_via_curl().

We leave out http-fetch and http-push, which don't use remotes for the
moment, purposefully.

Signed-off-by: Mike Hommey <mh@glandium.org>
---

 > > It's a good idea, but maybe http_init() (and the call chain leading up to
 > > it) should take the struct remote, so that it can get anything else you
 > > might also have there in the future? For that matter, is it intentional
 > > that http-push ignore the proxy setting, which is equally available in
 > > http-push, but not used? If so, it should probably get a comment.
 > 
 > It is intentional, for the moment. I'll rework the patch to take the
 > struct remote, and add a comment.

 And here you are. Changing const char * to struct remote * had the nice side
 effect to make me detect a missing change in builtin-http-fetch.c.

 builtin-http-fetch.c |    2 +-
 http-push.c          |    2 +-
 http-walker.c        |    4 ++--
 http.c               |   10 +++++++++-
 http.h               |    3 ++-
 transport.c          |    9 ++++-----
 walker.h             |    4 +++-
 7 files changed, 22 insertions(+), 12 deletions(-)

diff --git a/builtin-http-fetch.c b/builtin-http-fetch.c
index 7f450c6..48128c6 100644
--- a/builtin-http-fetch.c
+++ b/builtin-http-fetch.c
@@ -59,7 +59,7 @@ int cmd_http_fetch(int argc, const char **argv, const char *prefix)
 		url = rewritten_url;
 	}
 
-	walker = get_http_walker(url);
+	walker = get_http_walker(url, NULL);
 	walker->get_tree = get_tree;
 	walker->get_history = get_history;
 	walker->get_all = get_all;
diff --git a/http-push.c b/http-push.c
index 0beb740..04e056d 100644
--- a/http-push.c
+++ b/http-push.c
@@ -2240,7 +2240,7 @@ int main(int argc, char **argv)
 
 	memset(remote_dir_exists, -1, 256);
 
-	http_init();
+	http_init(NULL);
 
 	no_pragma_header = curl_slist_append(no_pragma_header, "Pragma:");
 
diff --git a/http-walker.c b/http-walker.c
index 2c37868..7bda34d 100644
--- a/http-walker.c
+++ b/http-walker.c
@@ -902,13 +902,13 @@ static void cleanup(struct walker *walker)
 	curl_slist_free_all(data->no_pragma_header);
 }
 
-struct walker *get_http_walker(const char *url)
+struct walker *get_http_walker(const char *url, struct remote *remote)
 {
 	char *s;
 	struct walker_data *data = xmalloc(sizeof(struct walker_data));
 	struct walker *walker = xmalloc(sizeof(struct walker));
 
-	http_init();
+	http_init(remote);
 
 	data->no_pragma_header = curl_slist_append(NULL, "Pragma:");
 
diff --git a/http.c b/http.c
index 519621a..64c53cd 100644
--- a/http.c
+++ b/http.c
@@ -219,13 +219,16 @@ static CURL* get_curl_handle(void)
 	return result;
 }
 
-void http_init(void)
+void http_init(struct remote *remote)
 {
 	char *low_speed_limit;
 	char *low_speed_time;
 
 	curl_global_init(CURL_GLOBAL_ALL);
 
+	if (remote && remote->http_proxy)
+		curl_http_proxy = xstrdup(remote->http_proxy);
+
 	pragma_header = curl_slist_append(pragma_header, "Pragma: no-cache");
 
 #ifdef USE_CURL_MULTI
@@ -315,6 +318,11 @@ void http_cleanup(void)
 
 	curl_slist_free_all(pragma_header);
 	pragma_header = NULL;
+
+	if (curl_http_proxy) {
+		free(curl_http_proxy);
+		curl_http_proxy = NULL;
+	}
 }
 
 struct active_request_slot *get_active_slot(void)
diff --git a/http.h b/http.h
index 9bab2c8..04169d5 100644
--- a/http.h
+++ b/http.h
@@ -7,6 +7,7 @@
 #include <curl/easy.h>
 
 #include "strbuf.h"
+#include "remote.h"
 
 /*
  * We detect based on the cURL version if multi-transfer is
@@ -83,7 +84,7 @@ extern void add_fill_function(void *data, int (*fill)(void *));
 extern void step_active_slots(void);
 #endif
 
-extern void http_init(void);
+extern void http_init(struct remote *remote);
 extern void http_cleanup(void);
 
 extern int data_received;
diff --git a/transport.c b/transport.c
index 397983d..86e0374 100644
--- a/transport.c
+++ b/transport.c
@@ -442,7 +442,8 @@ static struct ref *get_refs_via_curl(struct transport *transport)
 	struct ref *last_ref = NULL;
 
 	if (!transport->data)
-		transport->data = get_http_walker(transport->url);
+		transport->data = get_http_walker(transport->url,
+						transport->remote);
 
 	refs_url = xmalloc(strlen(transport->url) + 11);
 	sprintf(refs_url, "%s/info/refs", transport->url);
@@ -453,9 +454,6 @@ static struct ref *get_refs_via_curl(struct transport *transport)
 	curl_easy_setopt(slot->curl, CURLOPT_WRITEFUNCTION, fwrite_buffer);
 	curl_easy_setopt(slot->curl, CURLOPT_URL, refs_url);
 	curl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, NULL);
-	if (transport->remote->http_proxy)
-		curl_easy_setopt(slot->curl, CURLOPT_PROXY,
-				 transport->remote->http_proxy);
 
 	if (start_active_slot(slot)) {
 		run_active_slot(slot);
@@ -509,7 +507,8 @@ static int fetch_objs_via_curl(struct transport *transport,
 				 int nr_objs, struct ref **to_fetch)
 {
 	if (!transport->data)
-		transport->data = get_http_walker(transport->url);
+		transport->data = get_http_walker(transport->url,
+						transport->remote);
 	return fetch_objs_via_walker(transport, nr_objs, to_fetch);
 }
 
diff --git a/walker.h b/walker.h
index ea2c363..e1d40de 100644
--- a/walker.h
+++ b/walker.h
@@ -1,6 +1,8 @@
 #ifndef WALKER_H
 #define WALKER_H
 
+#include "remote.h"
+
 struct walker {
 	void *data;
 	int (*fetch_ref)(struct walker *, char *ref, unsigned char *sha1);
@@ -32,6 +34,6 @@ int walker_fetch(struct walker *impl, int targets, char **target,
 
 void walker_free(struct walker *walker);
 
-struct walker *get_http_walker(const char *url);
+struct walker *get_http_walker(const char *url, struct remote *remote);
 
 #endif /* WALKER_H */
-- 
1.5.4.3.195.g48d21.dirty

Re: [PATCH] Set proxy override with http_init()

[Daniel Barkalow]

On Wed, 27 Feb 2008, Mike Hommey wrote:

> In transport.c, proxy setting (the one from the remote conf) was set through
> curl_easy_setopt() call, while http.c already does the same with the
> http.proxy setting. We now just use this infrastructure instead, and make
> http_init() now take the struct remote as argument so that it can take the
> http_proxy setting from there, and any other property that would be added
> later.
> 
> At the same time, we make get_http_walker() take a struct remote argument
> too, and pass it to http_init(), which makes remote defined proxy be used
> for more than get_refs_via_curl().
> 
> We leave out http-fetch and http-push, which don't use remotes for the
> moment, purposefully.
> 
> Signed-off-by: Mike Hommey <mh@glandium.org>

Acked-by: Daniel Barkalow <barkalow@iabervon.org>

	-Daniel
*This .sig left intentionally blank*

RE: FW: git via http protocol _and_ a proxy using NTLM authentication -- git 1.5.4.2 & curl 7.18.0

[Ken.Fuchs]

> > $ GIT_CURL_VERBOSE=1 git clone \
> >   http://www.kernel.org/pub/scm/git/git.git
> > Initialized empty Git repository in /mnt/s2u1/git/test-git/git/.git/
> > Cannot get remote repository information.
> > Perhaps git-update-server-info needs to be run there?
> > $
 
> git-clone is not a builtin yet. When it is (and a patch has come for
> that), it will correctly use proxys.

Is there a work-around for cloning a git repository via a proxy?

Worse case, it should be possible to clone a repository on a
machine that doesn't require a proxy.  Convert the repository
into a tarball, copy it to the machine that requires a proxy
and change the config to work with the proxy, right?  At this
point, the git fetch command should work with the proxy, since
the local repository now has the required refs, right?

Thanks,

Ken Fuchs

RE: FW: git via http protocol _and_ a proxy using NTLM authentication -- git 1.5.4.2 & curl 7.18.0

[Johannes Schindelin]

Hi,

On Wed, 27 Feb 2008, Ken.Fuchs@bench.com wrote:

[Ken did not say who said this:]

> > git-clone is not a builtin yet. When it is (and a patch has come for
> > that), it will correctly use proxys.
> 
> Is there a work-around for cloning a git repository via a proxy?

Yes.

Just export "http_proxy".  AFAICT this works... See

http://repo.or.cz/w/msysgit.git?a=commitdiff;h=257a62710c0253d940a59b6fb2f371afa84dcf56

for a working example.

Hth,
Dscho

RE: FW: git via http protocol _and_ a proxy using NTLM authentication -- git 1.5.4.2 & curl 7.18.0

[Ken.Fuchs]

On Wed, 27 Feb 2008, Johannes Schindelin wrote:

> On Wed, 27 Feb 2008, Ken Fuchs wrote:
> 
> [Ken did not say who said this:]

Sorry, Mike Hommey wrote the following on 27 Feb 2008:

> > > git-clone is not a builtin yet. When it is (and a patch 
> > > has come for that), it will correctly use proxys.
> > 
> > Is there a work-around for cloning a git repository via a proxy?

> Yes.
> 
> Just export "http_proxy".  AFAICT this works... See
> 
>
http://repo.or.cz/w/msysgit.git?a=commitdiff;h=257a62710c0253d940a59b6fb
2f371afa84dcf56

> for a working example.

Unfortunately, this will not work because "git clone" is
implemented by git-clone.sh and has no support for dealing
with proxies at all.  Note that the example uses "git fetch"
and not "get clone".

So it seems that git's http protocol via a proxy isn't completely
supported.  For example, "git fetch" works with a proxy, since
it is implemented via a builtin (executable) which includes
proxy support.

At least that is my understanding of Mike's comments above.

The only work-around for "git clone" that I'm aware of is a
"proxy agent" that intercepts and modifies all Internet bound
requests and responses so they go (transparently) through the
proxy server rather than directly to the Internet.  Thus, none
of the clients being used need to be proxy aware, including
"git clone" (or any git function for that matter).

However, I was looking for a different work-around for
using "git clone" via http protocol and proxy (NTLM auth.),
but there do not appear to be any.

Thanks,

Ken Fuchs

RE: FW: git via http protocol _and_ a proxy using NTLM authentication -- git 1.5.4.2 & curl 7.18.0

[Johannes Schindelin]

Hi,

On Thu, 28 Feb 2008, Ken.Fuchs@bench.com wrote:

> On Wed, 27 Feb 2008, Johannes Schindelin wrote:
> 
> > On Wed, 27 Feb 2008, Ken Fuchs wrote:
> > 
> > [Ken did not say who said this:]
> 
> Sorry, Mike Hommey wrote the following on 27 Feb 2008:
> 
> > > > git-clone is not a builtin yet. When it is (and a patch 
> > > > has come for that), it will correctly use proxys.
> > > 
> > > Is there a work-around for cloning a git repository via a proxy?
> 
> > Yes.
> > 
> > Just export "http_proxy".  AFAICT this works... See
> > 
> >
> http://repo.or.cz/w/msysgit.git?a=commitdiff;h=257a62710c0253d940a59b6fb
> 2f371afa84dcf56
> 
> > for a working example.
> 
> Unfortunately, this will not work because "git clone" is
> implemented by git-clone.sh and has no support for dealing
> with proxies at all.

I fail to see why git-clone.sh, which uses git http-fetch directly, should 
behave differently from git fetch, which uses git http-fetch directly.

I strongly suspect that you did not even try what I suggested.

> So it seems that git's http protocol via a proxy isn't completely 
> supported.

If that should really be the case, well, Git is a programmers' tool, so 
you should be perfectly able to find out what is happening _exactly_.

If not, well, I heard subversion and Mercurial have nice Windows support.

Ciao,
Dscho

RE: FW: git via http protocol _and_ a proxy using NTLM authentication -- git 1.5.4.2 & curl 7.18.0

[Ken.Fuchs]

> On Thu, 28 Feb 2008, Ken.Fuchs@bench.com wrote:
> 
> > Unfortunately, this will not work because "git clone" is
> > implemented by git-clone.sh and has no support for dealing
> > with proxies at all.
> 
> I fail to see why git-clone.sh, which uses git http-fetch 
> directly, should behave differently from git fetch, which
> uses git http-fetch directly.

Sorry, I wasn't very clear.  When I said "git fetch" works, I meant
only the proxy is initiated properly.  "git fetch" itself still fails:

fatal: Couldn't find remote ref HEAD 

As I understand it, this is due to "git clone" failing earlier.

> I strongly suspect that you did not even try what I suggested.

I know you were trying to help.  I sincerely regret not thanking
you in my earlier response to your suggestion.  However, I had
already tried your suggestion over a week ago (plus numerous
user-id:password variants of the http_proxy environment variable).
I also tried "git config http.proxy <proxy.domain>:<proxy-port>".
I did not try "git config remote.<name>.proxy" which had an issue
that was recently fixed via a patch submission a few days ago.

> > So it seems that git's http protocol via a proxy isn't completely 
> > supported.
> 
> If that should really be the case, well, Git is a programmers' tool,
so 
> you should be perfectly able to find out what is happening _exactly_.

Well, all git http proxy stuff is handled by curl, so GIT_CURL_VERBOSE=1
should get us pretty close to "what is happening _exactly_":

$ GIT_CURL_VERBOSE=1 git clone http://www.kernel.org/pub/scm/git/git.git
Initialized empty Git repository in /mnt/nfs/git/test-git/git/.git/
Cannot get remote repository information.
Perhaps git-update-server-info needs to be run there?
$

No verbose output from curl?  How could that happen?  Maybe
GIT_CURL_VERBOSE=1 isn't logging all curl (especially proxy) actions?

So, let's try tcpdump to see what's happening on the wire.  In another
shell (logged in as root), we start:
"tcpdump -i eth0 | grep <proxy.domain>"

We should at least see DNS lookup requests in the tcpdump for the
<proxy.domain> if curl is using the proxy.

We repeat the above "git clone" command and find that absolutely no
proxy related traffic has been sent via eth0.  (The above tcpdump
pipe to grep generated no output whatsoever.)

On the other hand, I know that "git fetch" does properly interact with
the proxy, so as a benchmark, we repeat the tcpdump experiment with
"git fetch".  Now "tcpdump -i eth0 | grep <proxy.domain>" generates
plenty of output, consisting of DNS requests/responses for
<proxy.domain> and many packets between the git client and the proxy.

Also, the command

$ GIT_CURL_VERBOSE=1 git fetch http://www.kernel.org/pub/scm/git/git.git

generates plenty of curl logging information (indicating successful
proxy actions).  Since Bogofilter thinks this type of output is SPAM,
I won't try to include it all, but just the last 2 lines:

* Connection #0 to host <proxy.domain> left intact
fatal: Couldn't find remote ref HEAD

Again, as I understand it, "git fetch" (which is proxy aware) is
failing not due to proxy issues, but due to missing git filesystem
structure that "git clone" (which is not proxy aware) needed to provide.
If the necessary git filesystem structure existed in the local git
repository, the "git fetch" command would probably succeed via http
proxy.

Before closing, I'll note that http.c was patched slightly to allow
NTLM authentication via an http proxy server.  The informal patches
can be found earlier in this thread; they are clearly a
work-in-progress.
For example, proper user interface mechanisms (via "git config") still
need to be developed/incorporated.

Sincerely,

Ken Fuchs

Re: FW: git via http protocol _and_ a proxy using NTLM authentication -- git 1.5.4.2 & curl 7.18.0

[Mike Hommey]

On Fri, Feb 29, 2008 at 02:06:49PM -0600, Ken.Fuchs@bench.com wrote:
> > On Thu, 28 Feb 2008, Ken.Fuchs@bench.com wrote:
> > 
> > > Unfortunately, this will not work because "git clone" is
> > > implemented by git-clone.sh and has no support for dealing
> > > with proxies at all.
> > 
> > I fail to see why git-clone.sh, which uses git http-fetch 
> > directly, should behave differently from git fetch, which
> > uses git http-fetch directly.
> 
> Sorry, I wasn't very clear.  When I said "git fetch" works, I meant
> only the proxy is initiated properly.  "git fetch" itself still fails:
> 
> fatal: Couldn't find remote ref HEAD 
> 
> As I understand it, this is due to "git clone" failing earlier.
> 
> > I strongly suspect that you did not even try what I suggested.
> 
> I know you were trying to help.  I sincerely regret not thanking
> you in my earlier response to your suggestion.  However, I had
> already tried your suggestion over a week ago (plus numerous
> user-id:password variants of the http_proxy environment variable).
> I also tried "git config http.proxy <proxy.domain>:<proxy-port>".
> I did not try "git config remote.<name>.proxy" which had an issue
> that was recently fixed via a patch submission a few days ago.
> 
> > > So it seems that git's http protocol via a proxy isn't completely 
> > > supported.
> > 
> > If that should really be the case, well, Git is a programmers' tool,
> so 
> > you should be perfectly able to find out what is happening _exactly_.
> 
> Well, all git http proxy stuff is handled by curl, so GIT_CURL_VERBOSE=1
> should get us pretty close to "what is happening _exactly_":
> 
> $ GIT_CURL_VERBOSE=1 git clone http://www.kernel.org/pub/scm/git/git.git
> Initialized empty Git repository in /mnt/nfs/git/test-git/git/.git/
> Cannot get remote repository information.
> Perhaps git-update-server-info needs to be run there?
> $

git clone is a shell script and doesn't put all the wrap-up around curl.
GIT_CURL_VERBOSE is of no effect, and git http proxy settings won't work.
Only http_proxy environment variable will work.

You can try modifying git-clone and add --proxy-ntlm or --proxy-anyauth
to curl calls, and set the http_proxy environment variable.

Alternatively, you can try again with (work in progress) builtin-clone.c.
http://thread.gmane.org/gmane.comp.version-control.git/75070/focus=75126

Mike