Git 1.7.5 problem with HTTPS

Git 1.7.5 problem with HTTPS

[Dmitry Smirnov]

Hi,

I have problems with downloading Android code from android.googlesource.com.

The error says: fatal: branch stable is not signed

I was trying to figure out what happens and finally came to conclusion
that this is a problem of the git.

When I try to clone the git itself using https
(https://git.kernel.org/pub/scm/git/git.git) I'm getting the follwing
error:
warning: remote HEAD refers to nonexistent ref, unable to checkout.

If I use the http URL (http://git.kernel.org/pub/scm/git/git.git) I
can clone with no problems.

I was also considering that the problem is caused by proxy. But when I
tried to clone the same git source from another host via the same
proxy, it works pretty good. The difference is the git version: on the
first host it is 1.7.5.4 (comes with Ubuntu 11.10), on the second -
1.7.0.4

I was trying to collect some tcpdump and it shows the follwoing sequence
15	1.962132	X.X.X.X	Y.Y.Y.Y	HTTP	204	CONNECT git.kernel.org:443 HTTP/1.1
17	3.687364	Y.Y.Y.Y	X.X.X.X	HTTP	105	HTTP/1.0 200 Connection established
19	3.764793	X.X.X.X	Y.Y.Y.Y	TLSv1	208	Client Hello
21	3.815135	X.X.X.X	Y.Y.Y.Y	TLSv1	215	Ignored Unknown Record
23	4.045326	Y.Y.Y.Y	X.X.X.X	TLSv1	2239	Server Hello, Certificate,
Server Key Exchange, Server Hello Done
25	4.055059	Y.Y.Y.Y	X.X.X.X	TLSv1	73	Alert (Level: Fatal, Description:
Protocol Version)

As you can see, session seems terminated with fatal alert. It looks
like client tries to use TLS 1.2 but server accepts only TLS 1.0. But
this is just assumption.
Also, it is no clear what is that Ignored Unknown Record. But it
contains uncripted text: GET
/pub/scm/git/git.git/info/refs?service=git-upload-pack HTTP/1.1
User-Agent: git/1.7.5.4 Host: git.kernel.org Accept: */* Pragma:
no-cache

So, any suggesstions?

Dmitry

Re: Git 1.7.5 problem with HTTPS

[Shawn Pearce]

On Tue, Nov 15, 2011 at 05:52, Dmitry Smirnov <divis1969@gmail.com> wrote:
> I have problems with downloading Android code from android.googlesource.com.
>
> The error says: fatal: branch stable is not signed
>
> I was trying to figure out what happens and finally came to conclusion
> that this is a problem of the git.

Not likely. This is an error printed by the "repo" tool used by
Android. It typically indicates the repo command you are executing is
pointing to a URL that may be a local mirror and contain additional
patches in it that were not signed by me.

I would suggest starting over by downloading repo per [1] and using
that script to start the process.

[1] http://source.android.com/source/downloading.html

> When I try to clone the git itself using https
> (https://git.kernel.org/pub/scm/git/git.git) I'm getting the follwing
> error:
> warning: remote HEAD refers to nonexistent ref, unable to checkout.
>
> If I use the http URL (http://git.kernel.org/pub/scm/git/git.git) I
> can clone with no problems.

This may be a problem with the git.kernel.org HTTP server. It is
probably not a problem with Git itself.

Re: Git 1.7.5 problem with HTTPS

[Junio C Hamano]

Shawn Pearce <spearce@spearce.org> writes:

> On Tue, Nov 15, 2011 at 05:52, Dmitry Smirnov <divis1969@gmail.com> wrote:
>
>> If I use the http URL (http://git.kernel.org/pub/scm/git/git.git) I
>> can clone with no problems.
>
> This may be a problem with the git.kernel.org HTTP server. It is
> probably not a problem with Git itself.

Not likely to be an issue with git.kernel.org either. I am puzzled.

$ git ls-remote https://git.kernel.org/pub/scm/git/git.git |
  grep -e HEAD -e master
bc1bbe0c19a6ff39522b4fa3259f34150e308e1f        HEAD
bc1bbe0c19a6ff39522b4fa3259f34150e308e1f        refs/heads/master

$ rungit v1.7.5.4 ls-remote https://git.kernel.org/pub/scm/git/git.git |
  grep -e HEAD -e master
bc1bbe0c19a6ff39522b4fa3259f34150e308e1f        HEAD
bc1bbe0c19a6ff39522b4fa3259f34150e308e1f        refs/heads/master

The only thing that immediately comes to mind is the rare smart-http
breakage in 1.7.5 but that was plugged in 1.7.5.1, so...

Re: Git 1.7.5 problem with HTTPS

[Dmitry Smirnov]

2011/11/15 Shawn Pearce <spearce@spearce.org>:
> I would suggest starting over by downloading repo per [1] and using
> that script to start the process.
Cloning of the repo.git repository produces the same error
warning: remote HEAD refers to nonexistent ref, unable to checkout.

That is why I tried to use another repository...

Re: Git 1.7.5 problem with HTTPS

[Dmitry Smirnov]

2011/11/16 Junio C Hamano <gitster@pobox.com>:
> $ git ls-remote https://git.kernel.org/pub/scm/git/git.git |
>  grep -e HEAD -e master

in my case this command produce no output.

What if problem is caused by curl or TLS lib (libcurl-gnutls?) which
is used by my git? Is there any to log something from git-remote-https
?

Dmitry

Re: Git 1.7.5 problem with HTTPS

[Junio C Hamano]

Dmitry Smirnov <divis1969@gmail.com> writes:

> 2011/11/16 Junio C Hamano <gitster@pobox.com>:
>> $ git ls-remote https://git.kernel.org/pub/scm/git/git.git |
>>  grep -e HEAD -e master
>
> in my case this command produce no output.
>
> What if problem is caused by curl or TLS lib (libcurl-gnutls?) which
> is used by my git?

Yeah, this is sounding very likely that this is an issue at the SSL/TLS
layer underneath whatever Git speaks.

Re: Git 1.7.5 problem with HTTPS

[Tay Ray Chuan]

On Wed, Nov 16, 2011 at 3:11 PM, Dmitry Smirnov <divis1969@gmail.com> wrote:
> What if problem is caused by curl or TLS lib (libcurl-gnutls?) which
> is used by my git? Is there any to log something from git-remote-https
> ?

You can run git with GIT_CURL_VERBOSE set, like this

  GIT_CURL_VERBOSE=1 git ls-remote ...

-- 
Cheers,
Ray Chuan

Re: Git 1.7.5 problem with HTTPS

[Dmitry Smirnov]

[-- Attachment #1: Type: text/plain, Size: 746 bytes --]

Thanks.
I had collected two logs (for clone and ls-remote, attached).
Unfortunately, I cannot see, why problem occurs. The only indication is
* Connection #0 seems to be dead!

Is it possible that curl sends the request in plain text?
And according to tcpdump, why git/curl sends the request before Server Hello?



2011/11/16 Tay Ray Chuan <rctay89@gmail.com>:
> On Wed, Nov 16, 2011 at 3:11 PM, Dmitry Smirnov <divis1969@gmail.com> wrote:
>> What if problem is caused by curl or TLS lib (libcurl-gnutls?) which
>> is used by my git? Is there any to log something from git-remote-https
>> ?
>
> You can run git with GIT_CURL_VERBOSE set, like this
>
>  GIT_CURL_VERBOSE=1 git ls-remote ...
>
> --
> Cheers,
> Ray Chuan
>

[-- Attachment #2: clone.log --]
[-- Type: text/x-log, Size: 1899 bytes --]

dsmirnov@dsmirnov-ubuntu2:~/projects/tmp$ GIT_CURL_VERBOSE=1 GIT_TRACE=true git clone --verbose https://git.kernel.org/pub/scm/git/git.git
trace: built-in: git 'clone' '--verbose' 'https://git.kernel.org/pub/scm/git/git.git'
Cloning into git...
trace: run_command: 'git-remote-https' 'origin' 'https://git.kernel.org/pub/scm/git/git.git'
* Couldn't find host git.kernel.org in the .netrc file; using defaults
* About to connect() to proxy proxy.yyyyy.yy port 3128 (#0)
*   Trying Y.Y.Y.Y... * Connected to proxy.yyyyy.yy (Y.Y.Y.Y) port 3128 (#0)
* Establish HTTP proxy tunnel to git.kernel.org:443
> CONNECT git.kernel.org:443 HTTP/1.1
Host: git.kernel.org:443
User-Agent: git/1.7.5.4
Proxy-Connection: Keep-Alive
Pragma: no-cache

< HTTP/1.0 200 Connection established
< 
* Proxy replied OK to CONNECT request
* found 157 certificates in /etc/ssl/certs/ca-certificates.crt
> GET /pub/scm/git/git.git/info/refs?service=git-upload-pack HTTP/1.1
User-Agent: git/1.7.5.4
Host: git.kernel.org
Accept: */*
Pragma: no-cache

* Connection #0 to host proxy.yyyyy.yy left intact
* Couldn't find host git.kernel.org in the .netrc file; using defaults
* Connection #0 seems to be dead!
* Closing connection #0
* About to connect() to proxy proxy.yyyyy.yy port 3128 (#0)
*   Trying Y.Y.Y.Y... * Connected to proxy.yyyyy.yy (Y.Y.Y.Y) port 3128 (#0)
* Establish HTTP proxy tunnel to git.kernel.org:443
> CONNECT git.kernel.org:443 HTTP/1.1
Host: git.kernel.org:443
User-Agent: git/1.7.5.4
Proxy-Connection: Keep-Alive
Pragma: no-cache

< HTTP/1.0 200 Connection established
< 
* Proxy replied OK to CONNECT request
* found 157 certificates in /etc/ssl/certs/ca-certificates.crt
> GET /pub/scm/git/git.git/HEAD HTTP/1.1
User-Agent: git/1.7.5.4
Host: git.kernel.org
Accept: */*
Pragma: no-cache

* Connection #0 to host proxy.yyyyy.yy left intact
warning: remote HEAD refers to nonexistent ref, unable to checkout.

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #3: ls-remote.log --]
[-- Type: text/x-log; name="ls-remote.log", Size: 2365 bytes --]

dsmirnov@dsmirnov-ubuntu2:~/projects/tmp$ GIT_CURL_VERBOSE=1 GIT_TRACE=true git ls-remote https://git.kernel.org/pub/scm/git/git.git
trace: built-in: git 'ls-remote' 'https://git.kernel.org/pub/scm/git/git.git'
trace: run_command: 'git-remote-https' 'https://git.kernel.org/pub/scm/git/git.git' 'https://git.kernel.org/pub/scm/git/git.git'
* Couldn't find host git.kernel.org in the .netrc file; using defaults
* About to connect() to proxy proxy.yyyyy.yy port 3128 (#0)
*   Trying Y.Y.Y.Y... * Connected to proxy.yyyyy.yy (Y.Y.Y.Y) port 3128 (#0)
* Establish HTTP proxy tunnel to git.kernel.org:443
> CONNECT git.kernel.org:443 HTTP/1.1
Host: git.kernel.org:443
User-Agent: git/1.7.5.4
Proxy-Connection: Keep-Alive
Pragma: no-cache

< HTTP/1.0 200 Connection established
< 
* Proxy replied OK to CONNECT request
* found 157 certificates in /etc/ssl/certs/ca-certificates.crt
> GET /pub/scm/git/git.git/info/refs?service=git-upload-pack HTTP/1.1
User-Agent: git/1.7.5.4
Host: git.kernel.org
Accept: */*
Pragma: no-cache

* Connection #0 to host proxy.yyyyy.yy left intact
* Couldn't find host git.kernel.org in the .netrc file; using defaults
* Connection #0 seems to be dead!
* Closing connection #0
* About to connect() to proxy proxy.yyyyy.yy port 3128 (#0)
*   Trying Y.Y.Y.Y... * Connected to proxy.yyyyy.yy (Y.Y.Y.Y) port 3128 (#0)
* Establish HTTP proxy tunnel to git.kernel.org:443
> CONNECT git.kernel.org:443 HTTP/1.1
Host: git.kernel.org:443
User-Agent: git/1.7.5.4
Proxy-Connection: Keep-Alive
Pragma: no-cache

< HTTP/1.0 200 Connection established
< 
* Proxy replied OK to CONNECT request
* found 157 certificates in /etc/ssl/certs/ca-certificates.crt
> GET /pub/scm/git/git.git/HEAD HTTP/1.1
User-Agent: git/1.7.5.4
Host: git.kernel.org
Accept: */*
Pragma: no-cache

* Connection #0 to host proxy.yyyyy.yy left intact
0000000000000000000000000000000000000000	\x03\x01
0000000000000000000000000000000000000000	Cape
0000000000000000000000000000000000000000	\x06\x03U\x04\x06\x13\x02US1\x130\x11\x06\x03U\x13
ca00000000000000000000000000000000000000	\x06\x03U\x04\x06\x13\x02US1\x130\x11\x06\x03U\x13
\x01\x01\x010000000000000000000000000000000000000	*�H�
\x01\x01\x010000000000000000000000000000000000000	*�H�
0000000000000000000000000000000000000000	
0000000000000000000000000000000000000000	(��v�K\�M�d�����H���o\x10o\x19��\x1dD����]��g�\x02I\x01�Fo�1���﷋��_\x02^U#��2o�����

Re: Git 1.7.5 problem with HTTPS

[Daniel Stenberg]

On Wed, 16 Nov 2011, Dmitry Smirnov wrote:

> Unfortunately, I cannot see, why problem occurs. The only indication is
> * Connection #0 seems to be dead!

That means libcurl wanted to re-use an existing connection, but it seems to 
have died in the mean time and therefore it has to create a new one and 
reconnect instead. I suppose that is the first indication that something isn't 
quite right.

> Is it possible that curl sends the request in plain text?

I'd say that isn't very likely and you could easily snoop on the network to 
figure that out for sure.

> And according to tcpdump, why git/curl sends the request before Server 
> Hello?

curl will send the HTTP request once the TLS negotiation has completed as told 
by the TLS library. I believe you said you're using GnuTLS, are you using a 
recent version?

This is not a transfer layer (curl/HTTPS) bug I recognize, but I can of course 
not rule out that there's a bug somewhere in there!

-- 

  / daniel.haxx.se

Re: Git 1.7.5 problem with HTTPS

[Dmitry Smirnov]

> I'd say that isn't very likely and you could easily snoop on the network to> figure that out for sure

In the very first message I wrote that there is strange tcpdump record:
21      3.815135        X.X.X.X Y.Y.Y.Y TLSv1   215     Ignored Unknown Record
In this record there is some binary dump followed by an uncripted text:
GET /pub/scm/git/git.git/info/refs?service=git-upload-pack HTTP/1.1
User-Agent: git/1.7.5.4 Host: git.kernel.org Accept: */* Pragma:
no-cache

This packet is recorded before negotiation complete, so I'm wondering
who is guilty: git or curl?
What Git is providing to libcurl? Can I log it?

> curl will send the HTTP request once the TLS negotiation has completed as
> told by the TLS library. I believe you said you're using GnuTLS, are you
> using a recent version?
I'm using the version that comes with Ubuntu 11.10.

Re: Git 1.7.5 problem with HTTPS

[Haitao Li]

> I was also considering that the problem is caused by proxy. But when I
> tried to clone the same git source from another host via the same
> proxy, it works pretty good. The difference is the git version: on the
> first host it is 1.7.5.4 (comes with Ubuntu 11.10), on the second -
> 1.7.0.4

The proxy may have some impact.

I see exactly the same error only behind a proxy on my laptop running
Ubuntu 11.10 with libgnutls26/2.10.5-1ubuntu3. The same laptop works fine
at home without proxy.

I have another machine (Ubuntu 11.04 git/1.7.4.1 libgnutls26/2.8.6-1ubuntu2)
works fine behind the same proxy.

Re: Git 1.7.5 problem with HTTPS

[Jonathan Nieder]

Dmitry Smirnov wrote:

> What Git is providing to libcurl? Can I log it?

ltrace can help.

Re: Git 1.7.5 problem with HTTPS

[Dmitry Smirnov]

I had fixed the problem by manually installing the most recent version
of the libcurl3-gnutls for Ubuntu (from precise):
http://packages.ubuntu.com/precise/libcurl3-gnutls
It will require also most recent libgnutls:
http://packages.ubuntu.com/precise/libgnutls26

Dmitry