Default authentication over https?

Default authentication over https?

[Isaac Levy]

Hi all,

I use a git server which requires authentication over https. Git seems
determined to always try an unauthenticated request first, slowing
down operations by a couple seconds.

Is there a way to configure git to default to authenticated requests?  Thanks!

Regards,
Isaac Levy

Re: Default authentication over https?

[Jeff King]

On Mon, Apr 11, 2016 at 12:04:02PM -0400, Isaac Levy wrote:

> I use a git server which requires authentication over https. Git seems
> determined to always try an unauthenticated request first, slowing
> down operations by a couple seconds.
> 
> Is there a way to configure git to default to authenticated requests?  Thanks!

No, there isn't. Very old versions of git would ask for the password if
you provided a username, but since v1.7.8 we only do so in response to
an HTTP 401. The code is still there to do the "proactive" asking, but
it's not wired up to any particular config option.

However, I don't think even that would give you what you want. Because I
think that even if we provide a credential, curl will make an initial
request (presumably to find out which auth type it should use, but that
is just a guess). I don't know if there is a way to convince curl to
stick the credential in the first request (if my guess is right, then
perhaps by setting the auth type explicitly, or even by sticking in our
own Authorization header).

So I think the answer for now is "no", but it might be possible (and not
even that hard) to do with a patch.

-Peff

Re: Default authentication over https?

[Daniel Stenberg]

On Wed, 13 Apr 2016, Jeff King wrote:

> However, I don't think even that would give you what you want. Because I
> think that even if we provide a credential, curl will make an initial
> request (presumably to find out which auth type it should use, but that
> is just a guess). I don't know if there is a way to convince curl to
> stick the credential in the first request

curl supports this. but then you must do exactly that: tell libcurl to use 
that single auth method only. It will of course make it fail if you select the 
wrong method etc.

The unauthenticated first request is both to probe for which methods the 
server wants, but also works for the case when users provide credentials 
without the server actually ending up asking for them...

-- 

  / daniel.haxx.se

Re: Default authentication over https?

[Isaac Levy]

After the authenticated request, curl says it's keeping the connection
open, but the next fetch seems to do two handshakes again.  The
unauthenticated request closes the connection, so the 2nd handshake is
forced, but I'm not sure why subsequent git fetches still do
handshakes.  I did a bit of sleuthing w/ source, GIT_CURL_VERBOSE and
wireshark.

We're using GitHub enterprise -- it'd just be nice if there was a
better way to configure for super fast fetches.  ssh with cached
connections does avoid the SSL this overhead -- though as I recall git
protocols over ssh are less performant.

Finally I also checked out the persistent-https contrib section as a
workaround but couldn't get it to work. Is that project dead?

Thank you for your replies and support.

Regards,
Isaac

On Thu, Apr 14, 2016 at 5:46 AM, Daniel Stenberg <daniel@haxx.se> wrote:
> On Wed, 13 Apr 2016, Jeff King wrote:
>
>> However, I don't think even that would give you what you want. Because I
>> think that even if we provide a credential, curl will make an initial
>> request (presumably to find out which auth type it should use, but that
>> is just a guess). I don't know if there is a way to convince curl to
>> stick the credential in the first request
>
>
> curl supports this. but then you must do exactly that: tell libcurl to use
> that single auth method only. It will of course make it fail if you select
> the wrong method etc.
>
> The unauthenticated first request is both to probe for which methods the
> server wants, but also works for the case when users provide credentials
> without the server actually ending up asking for them...
>
> --
>
>  / daniel.haxx.se

Re: Default authentication over https?

[Jeff King]

On Thu, Apr 14, 2016 at 05:32:16PM -0400, Isaac Levy wrote:

> After the authenticated request, curl says it's keeping the connection
> open, but the next fetch seems to do two handshakes again.  The
> unauthenticated request closes the connection, so the 2nd handshake is
> forced, but I'm not sure why subsequent git fetches still do
> handshakes.  I did a bit of sleuthing w/ source, GIT_CURL_VERBOSE and
> wireshark.
> 
> We're using GitHub enterprise -- it'd just be nice if there was a
> better way to configure for super fast fetches.  ssh with cached
> connections does avoid the SSL this overhead -- though as I recall git
> protocols over ssh are less performant.

It's the opposite; generally git is more efficient over ssh, because we
have a true full-duplex connection, and we can do everything over a
single session (though setup for the ssh session may or may not be
faster than SSL).

Here's what I observed just for the initial contact. If I instrument git
like this:

diff --git a/http.c b/http.c
index 4304b80..9bedad7 100644
--- a/http.c
+++ b/http.c
@@ -1422,6 +1422,7 @@ static int http_request(const char *url,
 	curl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, headers);
 	curl_easy_setopt(slot->curl, CURLOPT_ENCODING, "gzip");
 
+	warning("running one curl slot...");
 	ret = run_one_slot(slot, &results);
 
 	if (options && options->content_type) {

and run:

  GIT_CURL_VERBOSE=1 git ls-remote 2>&1 | egrep '< HTTP|> |^Auth|^warn'

I get three requests:

  warning: running one curl slot...
  > GET /my/repo/info/refs?service=git-upload-pack HTTP/1.1
  < HTTP/1.1 401 Authorization Required
  warning: running one curl slot...
  > GET /my/repo/info/refs?service=git-upload-pack HTTP/1.1
  < HTTP/1.1 401 Authorization Required
  > GET /my/repo/info/refs?service=git-upload-pack HTTP/1.1
  Authorization: Basic ...
  < HTTP/1.1 200 OK

The first request is us (git) asking curl to hit the URL, and curl
returns the 401 from the server to us. At that point we'll prompt the
user (or the credential helper) for the password, and then we'll give
that to curl and ask it to make the request again. Curl will do the
probe with the 401, because we haven't set an auth type, and then follow
up (without returning the 401 to git) with the right credentials.

I think we can take that down to _two_ requests pretty easily. We know
in the very first request that the server told us something like:

  < WWW-Authenticate: Basic realm="GitHub"

but curl doesn't remember that. However, we should be able to pull it
out of the old request and feed it into the new one. That would save the
second request, which is just a probe.

But ideally we'd have this down to one request. I think we could do
something like this:

diff --git a/http.c b/http.c
index 9bedad7..7e5009d 100644
--- a/http.c
+++ b/http.c
@@ -870,6 +870,11 @@ struct active_request_slot *get_active_slot(void)
 #ifdef LIBCURL_CAN_HANDLE_AUTH_ANY
 	curl_easy_setopt(slot->curl, CURLOPT_HTTPAUTH, http_auth_methods);
 #endif
+
+	/* XXX should be configurable via http.* or whatever */
+	curl_easy_setopt(slot->curl, CURLOPT_HTTPAUTH, CURLAUTH_BASIC);
+	credential_fill(&http_auth);
+
 	if (http_auth.password || curl_empty_auth)
 		init_curl_http_auth(slot->curl);
 

My request then looks like:

  warning: running one curl slot...
  > GET /my/repo/info/refs?service=git-upload-pack HTTP/1.1
  Authorization: Basic ...
  < HTTP/1.1 200 OK

with just a single round-trip.

Obviously the hard-coding there is wrong, but the mental model here is
that the user would do something like:

  git config http.https://github.com.auth basic

to say "I know I want to do Basic auth for this host", and that would
trigger git to tell that to curl, and to prompt for the password
preemptively.  The obvious downside is that it requires manual
configuration, and some clue about "Basic" versus other auth types.

I didn't trace the protocol further, but I suspect that curl ends up
doing that probe for _each_ request, because each time we hand it only
the credential without telling it that we know the server wants "Basic"
auth.

So that tries to keep the number of requests down in the first place. As
far as reusing connections for the connections we _do_ make, I'm not
sure. I do see curl claiming to leave the connection up:

  * Connection #0 to host github.com left intact

but then we seem to make a separate one for the next request:

  * Hostname github.com was found in DNS cache
  *   Trying 192.30.252.120...
  * Connected to github.com (192.30.252.120) port 443 (#1)

I'm not sure why that is. We do get SSL reuse:

  * SSL re-using session ID
  * SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256

which is good. And then of the two requests curl makes later, the second
one has:

  * Re-using existing connection! (#1) with host github.com

which is good. So I'm not sure why we don't reuse connection #0 from the
very first request. Perhaps it's something in the way we are setting up
the curl handle.

> Finally I also checked out the persistent-https contrib section as a
> workaround but couldn't get it to work. Is that project dead?

I don't know. It certainly hasn't seen a lot of activity lately, but
Google folks might still be using it (I think the original impetus was
for fetching of many repos from the same server for the Android
project). I don't know that we've seen any mention of wide use on the
list.

-Peff

Re: Default authentication over https?

[Jeff King]

On Fri, Apr 15, 2016 at 06:21:20PM -0400, Jeff King wrote:

> I think we can take that down to _two_ requests pretty easily. We know
> in the very first request that the server told us something like:
> 
>   < WWW-Authenticate: Basic realm="GitHub"
> 
> but curl doesn't remember that. However, we should be able to pull it
> out of the old request and feed it into the new one. That would save the
> second request, which is just a probe.

Hmm. Looks like we already pull this out of the curl result for other
reasons, but we never feed it back in to the next request. So if I do
this:

diff --git a/http.c b/http.c
index 9bedad7..add9bf2 100644
--- a/http.c
+++ b/http.c
@@ -1132,6 +1132,8 @@ static int handle_curl_result(struct slot_results *results)
 			return HTTP_NOAUTH;
 		} else {
 #ifdef LIBCURL_CAN_HANDLE_AUTH_ANY
+			if (results->auth_avail)
+				http_auth_methods = results->auth_avail;
 			http_auth_methods &= ~CURLAUTH_GSSNEGOTIATE;
 #endif
 			return HTTP_REAUTH;

that drops my test case down to two requests: once to find out that we
need auth via the 401, and then we feed curl sufficient information to
do the followup in a single request (the GSSNEGOTIATE thing there is a
hack from 4dbe664, which we can ignore for now).

Interestingly, curl _does_ reuse the connection this time. I'm still not
sure why it didn't in the original case. But this means the whole thing
is happening over a single TCP session, which is good (and I didn't have
to change my config at all).

-Peff

Re: Default authentication over https?

[brian m. carlson]

[-- Attachment #1: Type: text/plain, Size: 1564 bytes --]

On Fri, Apr 15, 2016 at 06:43:35PM -0400, Jeff King wrote:
> Hmm. Looks like we already pull this out of the curl result for other
> reasons, but we never feed it back in to the next request. So if I do
> this:
> 
> diff --git a/http.c b/http.c
> index 9bedad7..add9bf2 100644
> --- a/http.c
> +++ b/http.c
> @@ -1132,6 +1132,8 @@ static int handle_curl_result(struct slot_results *results)
>  			return HTTP_NOAUTH;
>  		} else {
>  #ifdef LIBCURL_CAN_HANDLE_AUTH_ANY
> +			if (results->auth_avail)
> +				http_auth_methods = results->auth_avail;
>  			http_auth_methods &= ~CURLAUTH_GSSNEGOTIATE;
>  #endif
>  			return HTTP_REAUTH;
> 
> that drops my test case down to two requests: once to find out that we
> need auth via the 401, and then we feed curl sufficient information to
> do the followup in a single request (the GSSNEGOTIATE thing there is a
> hack from 4dbe664, which we can ignore for now).
> 
> Interestingly, curl _does_ reuse the connection this time. I'm still not
> sure why it didn't in the original case. But this means the whole thing
> is happening over a single TCP session, which is good (and I didn't have
> to change my config at all).

This looks fine from my point of view.  GSS-Negotiate and Digest are
going to require at least one round-trip because they need the token or
nonce to authenticate.  Basic shouldn't, though.
-- 
brian m. carlson / brian with sandals: Houston, Texas, US
+1 832 623 2791 | https://www.crustytoothpaste.net/~bmc | My opinion only
OpenPGP: https://keybase.io/bk2204

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 836 bytes --]