git https transport and wrong password

git https transport and wrong password

[Mikko Rapeli]

Hi,

I have a problem with git (1.7.9 and 1.8.2.357.gcc3e4eb) and https transport
to gerrit server (2.5.1-3-g719dfc7). I'm producing the problem on Cygwin but my
colleagues have same issue on Linux as well.

Gerrit server is matching corporate policies with single sign on, so after
three failed login attempts the account gets locked until a password reset.

Git amplifies this problem by asking for users password only once, and if
user made a typo git is still re-using the wrong password enough times to
get an account immediately locked.

I have client side logs with GIT_CURL_VERBOSE=1 but from intranet so can't
publish them directly. Here's roughly what the log shows:

---------------------------------------------------------------

$ GIT_CURL_VERBOSE=1 git fetch
...
> GET /gerrit/.../info/refs?service=git-upload-pack HTTP/1.1
...
< HTTP/1.1 401 Authorization Required
...

---------- I guess git prompts for password here. --------------

* Issue another request to this URL: 'https://..info/refs?service=git-upload-pack'
...
* Re-using existing connection! ...
...
* Server auth using Basic with user '...'
> GET /gerrit/.../info/refs?service=git-upload-pack HTTP/1.1
Authorization: Basic ...
...
< HTTP/1.1 401 Authorization Required
< Date: ...
* Authentication problem. Ignoring this.
...
* The requested URL returned error: 401
* Closing connection 0
...
* About to connect() to ...
...
* Connected to ...
...
* STATE: PROTOCONNECT => DO handle...
* Server auth using Basic with user '...'
> GET /gerrit/.../info/refs?service=git-upload-pack HTTP/1.1
Authorization: Basic ...
...
* STATE: DO => DO_DONE handle...
* STATE: DO_DONE => WAITPERFORM handle...
* STATE: WAITPERFORM => PERFORM handle...
...
< HTTP/1.1 302 Found
...
< Location: ...funnylongurl
...
* Ignoring the response-body
* Connection #1 to host ... left intact
* Issue another request to this URL: '...funnylongurl'
...
* Server auth using Basic with user '...'
> GET ...funnylongurl
Authorization: Basic ...
...
* The requested URL returned error: 500 Internal Server Error
* Closing connection 1
...
* About to connect()...
...
* Server auth using Basic with user '...'
> GET /gerrit/.../info/refs HTTP/1.1
Authorization: Basic ...
...
< HTTP/1.1 302 Found
< Date...
< Set-Cookie...
< Cache-Control: no-store
< Location: ...funnylongurl
...
* Re-using existing connection! (#2)...
> GET ...funnylongurl
...
* The requested URL returned error: 500 Internal Server Error
* Closing connection 2
...
error: The requested URL returned error: 500 Internal Server Error while accessing ...
fatal: HTTP request failed

---------------------------------------------------------------

Any idea what could be wrong here? Is git client really retrying with the
bad password?

Regards,

-Mikko

Re: git https transport and wrong password

[Mikko Rapeli]

On Tue, Apr 02, 2013 at 06:54:40PM +0300, Mikko Rapeli wrote:
> I have client side logs with GIT_CURL_VERBOSE=1 but from intranet so can't
> publish them directly. Here's roughly what the log shows:

Maybe this is simpler summary:

$ grep "HTTP\/1.1" log.txt
> GET ...info/refs?service=git-upload-pack
< HTTP/1.1 401 Authorization required

password prompt here, and ctrl-c does not work in Cygwin, sigh.

> GET ...info/refs?service=git-upload-pack
< HTTP/1.1 401 Authorization required
> GET ...info/refs?service=git-upload-pack
< HTTP/1.1 302 Found

account locked I presume

> GET longredirecturl
> GET ...info/refs
> HTTP/1.1 302 Found
> GET longredirecturl

I was not able reproduce this issue using curl directly to get the info/refs
page.

-Mikko

Re: git https transport and wrong password

[Jeff King]

On Tue, Apr 02, 2013 at 06:54:40PM +0300, Mikko Rapeli wrote:

> I have a problem with git (1.7.9 and 1.8.2.357.gcc3e4eb) and https transport
> to gerrit server (2.5.1-3-g719dfc7). I'm producing the problem on Cygwin but my
> colleagues have same issue on Linux as well.
> 
> Gerrit server is matching corporate policies with single sign on, so after
> three failed login attempts the account gets locked until a password reset.
> 
> Git amplifies this problem by asking for users password only once, and if
> user made a typo git is still re-using the wrong password enough times to
> get an account immediately locked.

Hmm. The sequence should be:

  - request, get 401
  - prompt user for password
  - retry request with password
  - if 401, die

IOW, we should make only a single request with the credential, and
immediately die afterwards. We do hit once to get the initial 401, but
we do not even provide a username, so unless the corporate policy is
locking out based on IP, it should not matter (and if it is, that shows
a fundamental misunderstanding about how a 401 is supposed to work).

But from your log, I see:

> ---------------------------------------------------------------
> 
> $ GIT_CURL_VERBOSE=1 git fetch
> ...
> > GET /gerrit/.../info/refs?service=git-upload-pack HTTP/1.1
> ...
> < HTTP/1.1 401 Authorization Required
> ...

Here's our first 401. OK.

> ---------- I guess git prompts for password here. --------------

Maybe...see below.

> * Server auth using Basic with user '...'
> > GET /gerrit/.../info/refs?service=git-upload-pack HTTP/1.1
> Authorization: Basic ...
> ...
> < HTTP/1.1 401 Authorization Required
> < Date: ...
> * Authentication problem. Ignoring this.
> ...
> * The requested URL returned error: 401

We get another 401. Now git should die. But it doesn't:

> * STATE: PROTOCONNECT => DO handle...
> * Server auth using Basic with user '...'
> > GET /gerrit/.../info/refs?service=git-upload-pack HTTP/1.1
> Authorization: Basic ...

It makes another request instead.

Weirdly, this does not result in a 401:

> * STATE: DO => DO_DONE handle...
> * STATE: DO_DONE => WAITPERFORM handle...
> * STATE: WAITPERFORM => PERFORM handle...
> ...
> < HTTP/1.1 302 Found
> ...
> < Location: ...funnylongurl
> ...
> * Ignoring the response-body
> * Connection #1 to host ... left intact
> * Issue another request to this URL: '...funnylongurl'
> ...
> * Server auth using Basic with user '...'
> > GET ...funnylongurl
> Authorization: Basic ...
> ...
> * The requested URL returned error: 500 Internal Server Error
> * Closing connection 1

We get redirected somewhere where we provide the (presumably wrong)
credential again. I do not think that is git's fault; the server asked
us to make the extra request. Is that part of the lockout procedure? If
it is not, it seems odd that the server would issue a redirect for a
bogus auth (shouldn't it just keep giving us 401?).

I do not know what is going on with the redirection there, but I have a
hunch on the extra auth round-trip.  What does your remote URL look
like? Does it have your username (e.g., https://user@host/project.git)?

I have noticed that if curl sees such a URL, it attempts to do a
password-less authentication itself, before even handing control back to
git. So my above sequence would become:

  1. git feeds URL to curl, who makes request
  2. we get a 401
  3. curl says "Oh, I have a username; let me try that" and re-requests
  4. we get another 401, because we need a password
  5. curl says "that didn't work" and hands control back to git
  6. git requests a password from the user and gives it to curl
  7. curl retries with the password, but it's wrong, so that results in
     a 401, too

At the end of it, we've now made _two_ failed requests for user X,
rather than one. I don't know if there's a way to tell curl not to try
the extra user-only round-trip. But you can strip the username out of
your URL to avoid it.

-Peff

Re: git https transport and wrong password

[Mikko Rapeli]

On Tue, Apr 02, 2013 at 03:28:45PM -0400, Jeff King wrote:
> We get redirected somewhere where we provide the (presumably wrong)
> credential again. I do not think that is git's fault; the server asked
> us to make the extra request. Is that part of the lockout procedure? If
> it is not, it seems odd that the server would issue a redirect for a
> bogus auth (shouldn't it just keep giving us 401?).

I think it is supposed to be a catch all failure mode without any
authentication but is just wrong/buggy. I'll try to debug these by
issuing curl commands step by step.

> I do not know what is going on with the redirection there, but I have a
> hunch on the extra auth round-trip.  What does your remote URL look
> like? Does it have your username (e.g., https://user@host/project.git)?

Yes, that's the giturl format I have.

> I have noticed that if curl sees such a URL, it attempts to do a
> password-less authentication itself, before even handing control back to
> git. So my above sequence would become:
> 
>   1. git feeds URL to curl, who makes request
>   2. we get a 401
>   3. curl says "Oh, I have a username; let me try that" and re-requests
>   4. we get another 401, because we need a password
>   5. curl says "that didn't work" and hands control back to git
>   6. git requests a password from the user and gives it to curl
>   7. curl retries with the password, but it's wrong, so that results in
>      a 401, too
> 
> At the end of it, we've now made _two_ failed requests for user X,
> rather than one. I don't know if there's a way to tell curl not to try
> the extra user-only round-trip. But you can strip the username out of
> your URL to avoid it.

It did seem like there was just one GET and 401 return before password
was promptet. I'll tripple check that.

Played around with command line curl a bit and at least it did the right
thing with a URL without username -- failed with 401 after single try --
and with URL without username but username provided -u 'username' which
succeeded or failed on single try based on password.

Don't know anything about curl but maybe git could parse the url for a
username and prompt for the password before the first 401 failure roundtrip
that's now in place. I guess most of this logic is in http.c.

-Mikko

Re: git https transport and wrong password

[Jeff King]

On Tue, Apr 02, 2013 at 10:47:51PM +0300, Mikko Rapeli wrote:

> Don't know anything about curl but maybe git could parse the url for a
> username and prompt for the password before the first 401 failure roundtrip
> that's now in place. I guess most of this logic is in http.c.

We used to do that but stopped, as curl might also be able to retrieve
the password from .netrc; the extra prompt was an annoyance to users
in this situation.

Now that we have the credential subsystem, I would recommend dropping
usernames from all git-over-http URLs, and either:

  1. Using a credential helper that supports secure long-term storage
     (osxkeychain, wincred, etc).

  2. Specifying the username to the credential subsystem explicitly, by
     putting something like:

       [credential "https://yourhost/"]
              username = yourusername

     in your git config.

Obviously (1) is nicer, but you may have corporate policies against
storing credentials. Or you may have a complicated single sign-on
procedure, where the password changes. In that case, I would still say
it is worth writing a custom helper script that can feed the temporary
credential to git.

-Peff

Re: git https transport and wrong password

[Mikko Rapeli]

On Tue, Apr 02, 2013 at 04:05:51PM -0400, Jeff King wrote:
> On Tue, Apr 02, 2013 at 10:47:51PM +0300, Mikko Rapeli wrote:
> 
> > Don't know anything about curl but maybe git could parse the url for a
> > username and prompt for the password before the first 401 failure roundtrip
> > that's now in place. I guess most of this logic is in http.c.
> 
> We used to do that but stopped, as curl might also be able to retrieve
> the password from .netrc; the extra prompt was an annoyance to users
> in this situation.

Ok, I think I've seen this before and ended up storing passwords in .netrc.

> Now that we have the credential subsystem, I would recommend dropping
> usernames from all git-over-http URLs, and either:
> 
>   1. Using a credential helper that supports secure long-term storage
>      (osxkeychain, wincred, etc).
> 
>   2. Specifying the username to the credential subsystem explicitly, by
>      putting something like:
> 
>        [credential "https://yourhost/"]
>               username = yourusername
> 
>      in your git config.
> 
> Obviously (1) is nicer, but you may have corporate policies against
> storing credentials. Or you may have a complicated single sign-on
> procedure, where the password changes. In that case, I would still say
> it is worth writing a custom helper script that can feed the temporary
> credential to git.

Thanks, I'll have a look at these helpers. Policies we may have but in
practice I think many just store plaintext passwords in giturls, which
is obviously the worst case, but it works against accidental typos in
the password prompt (though blows up when the mandatory password change
comes along).

-Mikko

Re: git https transport and wrong password

[Mikko Rapeli]

Maybe my git installation was incomplete before when running from ~/bin since
I was not able to set break points to http_request() and some debug code
was not there until I ran git through bin-wrappers in the source tree.

I added some debug prints to http.c functions http_request() and
handle_curl_result(), and now I see this chain of events:

 http_request_reauth()
 http_request()
 GET ...info/refs?service=git-upload-pack
 HTTP/1.1 401 Authorization Required
* Ignoring the response-body
* Issue another request to this URL: '...'
 GET ...info/refs?service=git-upload-pack
 HTTP/1.1 401 Authorization Required
 handle_curl_result: res = 22, http_code = 401, user = ..., pass = (null)
Password for '...': (enter valid password)
 GET ...info/refs?service=git-upload-pack
 HTTP/1.1 200 OK

So, for some reason the first GET request is issued twice and first 401
is ignored. I'll try to debug run_active_slot() next...

-Mikko

Re: git https transport and wrong password

[Jeff King]

[+cc Daniel for curl questions below]

On Wed, Apr 03, 2013 at 12:43:02PM +0300, Mikko Rapeli wrote:

> Maybe my git installation was incomplete before when running from ~/bin since
> I was not able to set break points to http_request() and some debug code
> was not there until I ran git through bin-wrappers in the source tree.

Debugging git-over-http is somewhat difficult because the interesting
bits happen in sub-processes. You can get much closer to the http calls
by running the transport helper directly, like:

  gdb --args git-remote-https https://yourhost/

which will start by reading commands from stdin (try "list" to get it to
fetch the remote refs).

> I added some debug prints to http.c functions http_request() and
> handle_curl_result(), and now I see this chain of events:
> 
>  http_request_reauth()
>  http_request()
>  GET ...info/refs?service=git-upload-pack
>  HTTP/1.1 401 Authorization Required
> * Ignoring the response-body
> * Issue another request to this URL: '...'
>  GET ...info/refs?service=git-upload-pack
>  HTTP/1.1 401 Authorization Required
>  handle_curl_result: res = 22, http_code = 401, user = ..., pass = (null)
> Password for '...': (enter valid password)
>  GET ...info/refs?service=git-upload-pack
>  HTTP/1.1 200 OK
> 
> So, for some reason the first GET request is issued twice and first 401
> is ignored. I'll try to debug run_active_slot() next...

Right, I think that's curl trying to make use of the username in the
URL. Try this (I'm using github here as a convenient http servers, but
you should be able to replicate with your internal server):

  $ GIT_CURL_VERBOSE=1 git ls-remote https://foo@github.com/requires/auth \
      2>&1 >/dev/null | egrep '^>|^< HTTP|^Authorization|requested URL'
  > GET /requires/auth/info/refs?service=git-upload-pack HTTP/1.1
  < HTTP/1.1 401 Authorization Required
  > GET /requires/auth/info/refs?service=git-upload-pack HTTP/1.1
  Authorization: Basic Zm9vOg==
  < HTTP/1.1 401 Authorization Required
  * The requested URL returned error: 401
  Password for 'https://foo@github.com': 
  > GET /requires/auth/info/refs?service=git-upload-pack HTTP/1.1
  Authorization: Basic Zm9vOmJhcg==
  < HTTP/1.1 401 Authorization Required
  * The requested URL returned error: 401

So you can see that curl makes _two_ requests internally before it
returns the 401. One unadorned, and one with just the username
("Zm9vOg==", which decodes to "foo:") for the auth. Then git prompts for
the password, and we retry (and of course I am feeding it a bogus
username/password combo, so we get another 401).

I would expect without the username in the URL for it to make only two
requests: one to get the first 401, then git collects the credentials,
then a follow-up with the credentials. But instead we get:

  $ GIT_CURL_VERBOSE=1 git ls-remote https://github.com/requires/auth \
      2>&1 >/dev/null | egrep '^>|^< HTTP|^Authorization|requested URL'
  > GET /requires/auth/info/refs?service=git-upload-pack HTTP/1.1
  * The requested URL returned error: 401 Authorization Required
  Username for 'https://github.com': foo
  Password for 'https://foo@github.com': 
  > GET /requires/auth/info/refs?service=git-upload-pack HTTP/1.1
  < HTTP/1.1 401 Authorization Required
  > GET /requires/auth/info/refs?service=git-upload-pack HTTP/1.1
  Authorization: Basic Zm9vOmJhcg==
  < HTTP/1.1 401 Authorization Required
  * The requested URL returned error: 401

So we get a 401, as expected, git prompts for the credentials and feeds
them directly to curl, but then we still get _two_ requests: we trigger
another 401, and only then does curl provide the authorization header to
the server.

I'm not sure if that extra auth is intended or not.

It's also possible that git is screwing up in providing the credentials
to curl, but I don't think so. We feed them to the curl handle as soon
as we get them, and there should be only one handle in use here.

-Peff

Re: git https transport and wrong password

[Mikko Rapeli]

On Wed, Apr 03, 2013 at 10:12:12AM -0400, Jeff King wrote:
> I would expect without the username in the URL for it to make only two
> requests: one to get the first 401, then git collects the credentials,
> then a follow-up with the credentials. But instead we get:
> 
>   $ GIT_CURL_VERBOSE=1 git ls-remote https://github.com/requires/auth \
>       2>&1 >/dev/null | egrep '^>|^< HTTP|^Authorization|requested URL'
>   > GET /requires/auth/info/refs?service=git-upload-pack HTTP/1.1
>   * The requested URL returned error: 401 Authorization Required
>   Username for 'https://github.com': foo
>   Password for 'https://foo@github.com': 
>   > GET /requires/auth/info/refs?service=git-upload-pack HTTP/1.1
>   < HTTP/1.1 401 Authorization Required
>   > GET /requires/auth/info/refs?service=git-upload-pack HTTP/1.1
>   Authorization: Basic Zm9vOmJhcg==
>   < HTTP/1.1 401 Authorization Required
>   * The requested URL returned error: 401
> 
> So we get a 401, as expected, git prompts for the credentials and feeds
> them directly to curl, but then we still get _two_ requests: we trigger
> another 401, and only then does curl provide the authorization header to
> the server.
> 
> I'm not sure if that extra auth is intended or not.

git uses CURLAUTH_ANY which means: first try without authentication
(CURLAUTH_NONE), if that fails it will try (I guess) CURLAUTH_BASIC|DIGEST|
GSS|NTML and so on, and only then it will fail with the 401.

It seems that skipping CURLAUTH_NONE try is not possible even if it's
not a good idea when a username and possibly password is available.
Changing CURLAUTH_ANY to skip CURLAUTH_NONE could also break other
users.

Since netrc support really needs this one try from git to curl before
password prompt I guess in our case using HTTPS with git is simply not
feasible. Changing the corporate single sign-on policies is also hard
so I will now try to get SSH transport running on the server.

Account locking will still be quite easy but hopefully only after
multiple false passwords to the SSH promp.

-Mikko