Errors cloning over http -- git-clone and cg-clone fail to fetch a reachable object...

Errors cloning over http -- git-clone and cg-clone fail to fetch a reachable object...

[Martin Langhoff]

Strange!

I'm getting errors when cloning over http

git-clone http://locke.catalyst.net.nz/git/moodle.git mdlfoo
(...)
error:  (curl_result = 3601440, http_code = 200, sha1 =
f04241b142edfbf28fff2babb426cbab5b44e26b)
Getting pack list
error:
Getting alternates list
error: Unable to find f04241b142edfbf28fff2babb426cbab5b44e26b under
http://locke.catalyst.net.nz/git/moodle.git/

Cannot obtain needed commit f04241b142edfbf28fff2babb426cbab5b44e26b
while processing commit 0965f28d4d75f324b86c8f7490830fea471c65c5.

This commit object is easily reachable at
http://mirrors.catalyst.net.nz/git/moodle.git/objects/f0/4241b142edfbf28fff2babb426cbab5b44e26b

If I use cg-clone, I get a similar error

cg-clone http://locke.catalyst.net.nz/git/moodle.git#mdl-artena-tairawhiti
mdlfooo
(...)
Cannot obtain needed object 214e6374d49e6d014f0ba6f159d585a3fe468909
while processing commit 0000000000000000000000000000000000000000.
cg-fetch: objects fetch failed
cg-clone: fetch failed

This commit object seems to be in a pack:
http://mirrors.catalyst.net.nz/git/moodle.git/objects/pack/pack-094560c0177ad659a6e172739c4be53da749e5f0.pack

git-cat-file on the server works correctly, and cloning/working over
git+ssh works too.

cheers,


martin

Re: Errors cloning over http -- git-clone and cg-clone fail to fetch a reachable object...

[Martin Langhoff]

On 11/7/05, Martin Langhoff <martin.langhoff@gmail.com> wrote:
> Strange!
>
> I'm getting errors when cloning over http

More info on this.

git-fetch-pack (invoked by cg-fetch) bails out because it thinks it
got a 404 fetching one of the packs:

Getting pack 9cbe4a5eb777d4ee535f08feb471e812208ed3a5
 which contains 7004cdf821ab5ddcded7819dea34015b0e84cd9a
error: Unable to get pack file
http://locke.catalyst.net.nz/git/moodle.git//objects/pack/pack-9cbe4a5eb777d4ee535f08feb471e812208ed3a5.pack
The requested URL returned error: 404

However, the url is reachable via http (tested with curl and wget) and
Apache records the transaction as a 200 OK -- there's no 404 there!
(There are 404s, of course, for objects that are in the pack but not
for any pack). Trying to read http-fetch.c and http-pull.c to figure
out where we could get the return status wrong, but my C is just
nonexistant. Hints appreciated.

cheers,


martin

Re: Errors cloning over http -- git-clone and cg-clone fail to fetch a reachable object...

[Nick Hengeveld]

On Mon, Nov 07, 2005 at 03:52:41PM +1300, Martin Langhoff wrote:

> > I'm getting errors when cloning over http
> 
> More info on this.
> 
> git-fetch-pack (invoked by cg-fetch) bails out because it thinks it
> got a 404 fetching one of the packs:

Does it always error out on the same pack file?

Which build of git are you using, and which version of curl?

-- 
For a successful technology, reality must take precedence over public
relations, for nature cannot be fooled.

Re: Errors cloning over http -- git-clone and cg-clone fail to fetch a reachable object...

[Martin Langhoff]

On 11/7/05, Nick Hengeveld <nickh@reactrix.com> wrote:
> On Mon, Nov 07, 2005 at 03:52:41PM +1300, Martin Langhoff wrote:
>
> > > I'm getting errors when cloning over http
> >
> > More info on this.
> >
> > git-fetch-pack (invoked by cg-fetch) bails out because it thinks it
> > got a 404 fetching one of the packs:
>
> Does it always error out on the same pack file?

Yes.

> Which build of git are you using, and which version of curl?

A mix of 0.99.9, 0.99.9{b,c,d} and today's. Tested with Johannes'
patch of today for curl message handling, thinking it was related. No
dice.

On the Debian x86 box, it's
$ curl -V
curl 7.13.2 (i386-pc-linux-gnu) libcurl/7.14.0 OpenSSL/0.9.7g
zlib/1.2.2 libidn/0.5.18
Protocols: ftp gopher telnet dict ldap http file https ftps
Features: IDN IPv6 Largefile NTLM SSL libz

On the MacOSX box, it's
$ curl -V
curl 7.10.2 (powerpc-apple-darwin7.0) libcurl/7.10.2 OpenSSL/0.9.7g zlib/1.1.4

cheers,



martin

Re: Errors cloning over http -- git-clone and cg-clone fail to fetch a reachable object...

[Nick Hengeveld]

On Mon, Nov 07, 2005 at 05:50:02PM +1300, Martin Langhoff wrote:

> > Does it always error out on the same pack file?
> 
> Yes.

You might try this to see exactly what request/response headers
curl thinks are passing back and forth.


diff --git a/http-fetch.c b/http-fetch.c
index ea8af1b..11d4dca 100644
--- a/http-fetch.c
+++ b/http-fetch.c
@@ -325,6 +325,7 @@ static struct active_request_slot *get_a
 	curl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, pragma_header);
 	curl_easy_setopt(slot->curl, CURLOPT_HTTPHEADER, no_range_header);
 	curl_easy_setopt(slot->curl, CURLOPT_ERRORBUFFER, curl_errorstr);
+	curl_easy_setopt(slot->curl, CURLOPT_VERBOSE, 1);
 
 	return slot;
 }

-- 
For a successful technology, reality must take precedence over public
relations, for nature cannot be fooled.

Re: Errors cloning over http -- git-clone and cg-clone fail to fetch a reachable object...

[Martin Langhoff]

On 11/8/05, Nick Hengeveld <nickh@reactrix.com> wrote:

> You might try this to see exactly what request/response headers
> curl thinks are passing back and forth.

It definitely looks like it's doing a few requests in parallel and
getting them mixed up. BTW, this repo is public and sitting on a box
that doubles up as kernel mirror -- feel free to hit it ;-)

> GET /git/moodle.git/objects/f0/6a06d24eee0b7819e2aaf48ad0e255301394e0 HTTP/1.1
Host: locke.catalyst.net.nz
Accept: */*

* Connected to locke.catalyst.net.nz (202.78.240.39) port 80
* Couldn't find host locke.catalyst.net.nz in the .netrc file, using defaults
* About to connect() to locke.catalyst.net.nz port 80
*   Trying 202.78.240.39... > GET
/git/moodle.git/objects/d9/6d5ee03a225ab4e750fb864dbea35d42c51b8b
HTTP/1.1
Host: locke.catalyst.net.nz
Accept: */*

* Connected to locke.catalyst.net.nz (202.78.240.39) port 80
* The requested URL returned error: 404
* Closing connection #0
* The requested URL returned error: 404
* Closing connection #0
> GET /git/moodle.git/objects/5e/0e0d41f781d53344fa67b5e5a0138b586e2946 HTTP/1.1
Host: locke.catalyst.net.nz
Accept: */*

* The requested URL returned error: 404
* Closing connection #0
* The requested URL returned error: 404
* Closing connection #0
error: Unable to get pack file
http://locke.catalyst.net.nz/git/moodle.git//objects/pack/pack-9cbe4a5eb777d4ee535f08feb471e812208ed3a5.pack
The requested URL returned error: 404
error: Unable to find 7004cdf821ab5ddcded7819dea34015b0e84cd9a under
http://locke.catalyst.net.nz/git/moodle.git/

Cannot obtain needed blob 7004cdf821ab5ddcded7819dea34015b0e84cd9a
while processing commit b065a5cb7f757dd6e271249cb49e19e8c34b26ce.
cg-fetch: objects fetch failed
cg-clone: fetch failed

cheers,


martin

Re: Errors cloning over http -- git-clone and cg-clone fail to fetch a reachable object...

[Nick Hengeveld]

On Tue, Nov 08, 2005 at 03:37:29PM +1300, Martin Langhoff wrote:

> It definitely looks like it's doing a few requests in parallel and
> getting them mixed up. BTW, this repo is public and sitting on a box
> that doubles up as kernel mirror -- feel free to hit it ;-)

Can you run git-http-fetch after the failure to attempt to pick up where
it left off?  It should fail right away, and hopefully contain less
confusing output from parallel requests.

Was there a request header for pack-9cbe...d3a5.pack earlier in the
output, and were there response headers for successful or failed
requests?

FWIW, I've tried cloning that repository a few times and haven't seen
the problem yet.

-- 
For a successful technology, reality must take precedence over public
relations, for nature cannot be fooled.

Re: Errors cloning over http -- git-clone and cg-clone fail to fetch a reachable object...

[Johannes Schindelin]

Hi,

On Tue, 8 Nov 2005, Martin Langhoff wrote:

> On 11/8/05, Nick Hengeveld <nickh@reactrix.com> wrote:
> 
> > You might try this to see exactly what request/response headers
> > curl thinks are passing back and forth.
> 
> It definitely looks like it's doing a few requests in parallel and
> getting them mixed up. BTW, this repo is public and sitting on a box
> that doubles up as kernel mirror -- feel free to hit it ;-)

It could be a similar problem to what I experienced in http-push: 
Uninintialized memory due to the code path. Can you run

	valgrind --leak-check=yes --show-reachable=yes git-http-fetch...

and check for jump depending on uninitialized memory?

Hth,
Dscho

Re: Errors cloning over http -- git-clone and cg-clone fail to fetch a reachable object...

[Ben Clifford]

I got similar today; www.hawaga.org.uk is some apache server on  
linux, and piva.hawaga.org.uk, the client Mac OS X.

Feel free to hit the URL yourselves -- its a public webserver.


According to system profiler on piva:

   System Version:    Mac OS X 10.4.2 (8D37)
   Kernel Version:    Darwin 8.2.1

and

!582 [1] benc@piva:~/tmp$ curl --version
curl 7.13.1 (powerpc-apple-darwin8.0) libcurl/7.13.1 OpenSSL/0.9.7g  
zlib/1.2.3
Protocols: ftp gopher telnet dict ldap http file https ftps
Features: IPv6 Largefile NTLM SSL libz
!583 [1] benc@piva:~/tmp$

This is on an unpacked repository, so I would guess that none of the  
GETs for objects should be failing...

Note that it *doesn't* always error out on the same object:

!549 [0] benc@piva:~/tmp$ cg clone http://www.hawaga.org.uk/ 
gitcompletion.gitdefaulting to local storage area
20:56:33 URL:http://www.hawaga.org.uk/gitcompletion.git/HEAD [41/41] - 
 > "refs/heads/.origin-fetching" [1]
progress: 11 objects, 2875 bytes
error:  (curl_result = 3181280, http_code = 200, sha1 =  
c99aa418704f576aad8249d042cd6afecf38afc4)
Getting pack list
error:
Getting alternates list
error: Unable to find c99aa418704f576aad8249d042cd6afecf38afc4 under  
http://www.hawaga.org.uk/gitcompletion.git/

Cannot obtain needed blob c99aa418704f576aad8249d042cd6afecf38afc4
while processing commit 912eb2baf57b70178c2e1f10df12eae911e1d748.
cg-fetch: objects fetch failed
cg-clone: fetch failed
!550 [0] benc@piva:~/tmp$



!556 [0] benc@piva:~/tmp$ cg clone http://www.hawaga.org.uk/ 
gitcompletion.git
defaulting to local storage area
20:58:48 URL:http://www.hawaga.org.uk/gitcompletion.git/HEAD [41/41] - 
 > "refs/heads/.origin-fetching" [1]
progress: 12 objects, 4581 bytes
error:  (curl_result = 3179088, http_code = 200, sha1 =  
682cfbd4d75e204fada2cabe9e0f040e522ce61d)
Getting pack list
Getting alternates list
error: Unable to find 682cfbd4d75e204fada2cabe9e0f040e522ce61d under  
http://www.hawaga.org.uk/gitcompletion.git/

Cannot obtain needed commit 682cfbd4d75e204fada2cabe9e0f040e522ce61d
while processing commit 912eb2baf57b70178c2e1f10df12eae911e1d748.
Waiting for http://www.hawaga.org.uk/gitcompletion.git/objects/cb/ 
c8285a736142b83894ec412fe413d9ee7363ab
cg-fetch: objects fetch failed
cg-clone: fetch failed
!557 [0] benc@piva:~/tmp$

When I run it lots of times in succession, it gets between 10 and 15  
objects in before failing on something.

When I try with plain git:

!580 [1] benc@piva:~/tmp$ git clone http://www.hawaga.org.uk/ 
gitcompletion.git here
defaulting to local storage area
got a108bdc110dad770ec5c092759a8bc511790d21f
walk a108bdc110dad770ec5c092759a8bc511790d21f
got 912eb2baf57b70178c2e1f10df12eae911e1d748
got 261466f44dce8fb3cac4e0d0f1bf5a46fa84e07e
walk 912eb2baf57b70178c2e1f10df12eae911e1d748
got f766b276eff3ec52ac3c0425b13c936d87c607ee
got e05988e4a193f9eac2862ce6e67b19e3785bd8f1
got c99aa418704f576aad8249d042cd6afecf38afc4
got d743b178b82e9a8f10a1bf267d32d070e36c8046
got 96a0d6b6b182d978c04290f7c87049a5d3cc41b7
got 676bd846066cf3794796cadb258a23a76744819c
got 2e04450bdc81b1380a8f75223d8ce8019400dd70
error:  (curl_result = 3179232, http_code = 200, sha1 =  
682cfbd4d75e204fada2cabe9e0f040e522ce61d)
Getting pack list
error:
Getting alternates list
error: Unable to find 682cfbd4d75e204fada2cabe9e0f040e522ce61d under  
http://www.hawaga.org.uk/gitcompletion.git/

Cannot obtain needed commit 682cfbd4d75e204fada2cabe9e0f040e522ce61d
while processing commit 912eb2baf57b70178c2e1f10df12eae911e1d748.
!581 [1] benc@piva:~/tmp$


but a curl by hand for one of the object URLs works:

!581 [1] benc@piva:~/tmp$ curl http://www.hawaga.org.uk/ 
gitcompletion.git/objects/68/2cfbd4d75e204fada2cabe9e0f040e522ce61d  
 > /dev/null
   % Total    % Received % Xferd  Average Speed   Time    Time      
Time  Current
                                  Dload  Upload   Total   Spent     
Left  Speed
100   250  100   250    0     0    183      0  0:00:01  0:00:01  
--:--:--     0
!582 [1] benc@piva:~/tmp$


and works each time when I run it repeatedly.






On 7 Nov 2005, at 08:54, Martin Langhoff wrote:


> Strange!
>
> I'm getting errors when cloning over http
>
> git-clone http://locke.catalyst.net.nz/git/moodle.git mdlfoo
> (...)
> error:  (curl_result = 3601440, http_code = 200, sha1 =
> f04241b142edfbf28fff2babb426cbab5b44e26b)
> Getting pack list
> error:
> Getting alternates list
> error: Unable to find f04241b142edfbf28fff2babb426cbab5b44e26b under
> http://locke.catalyst.net.nz/git/moodle.git/
>
> Cannot obtain needed commit f04241b142edfbf28fff2babb426cbab5b44e26b
> while processing commit 0965f28d4d75f324b86c8f7490830fea471c65c5.
>
> This commit object is easily reachable at
> http://mirrors.catalyst.net.nz/git/moodle.git/objects/ 
> f0/4241b142edfbf28fff2babb426cbab5b44e26b
>
> If I use cg-clone, I get a similar error
>
> cg-clone http://locke.catalyst.net.nz/git/moodle.git#mdl-artena- 
> tairawhiti
> mdlfooo
> (...)
> Cannot obtain needed object 214e6374d49e6d014f0ba6f159d585a3fe468909
> while processing commit 0000000000000000000000000000000000000000.
> cg-fetch: objects fetch failed
> cg-clone: fetch failed
>
> This commit object seems to be in a pack:
> http://mirrors.catalyst.net.nz/git/moodle.git/objects/pack/ 
> pack-094560c0177ad659a6e172739c4be53da749e5f0.pack
>
> git-cat-file on the server works correctly, and cloning/working over
> git+ssh works too.
>
> cheers,
>
>
> martin
> -
> To unsubscribe from this list: send the line "unsubscribe git" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>
>

-- 
Ben ベン Бэн
http://www.hawaga.org.uk/ben/

Re: Errors cloning over http -- git-clone and cg-clone fail to fetch a reachable object...

[Nick Hengeveld]

On Tue, Nov 08, 2005 at 10:19:53PM +1100, Ben Clifford wrote:

> I got similar today; www.hawaga.org.uk is some apache server on  
> linux, and piva.hawaga.org.uk, the client Mac OS X.
> ...
> error:  (curl_result = 3181280, http_code = 200, sha1 =  
> c99aa418704f576aad8249d042cd6afecf38afc4)

Those curl result codes all look wrong, and sounds like a memory issue
that Johannes Schindelin recently fixed in commit
90279074ca5cc336a8bfffd47d19d089b291b432.  Does your git build have that
patch?

-- 
For a successful technology, reality must take precedence over public
relations, for nature cannot be fooled.

Re: Errors cloning over http -- git-clone and cg-clone fail to fetch a reachable object...

[Ben Clifford]

On 9 Nov 2005, at 12:09, Nick Hengeveld wrote:
>
> Those curl result codes all look wrong, and sounds like a memory issue
> that Johannes Schindelin recently fixed in commit
> 90279074ca5cc336a8bfffd47d19d089b291b432.  Does your git build have  
> that
> patch?

I did not - I've pulled and rebuilt from master, and have that commit  
now.
It works better (not perfectly (there's a tags 403) but I suspect  
that may be permissions config at my server end or absence of tags in  
my repo or something mumble). I can do this a bunch of times in a row  
with the same results and the resulting cloned repository looks sane.

Ben

!535 [0] benc@piva:~/tmp/xa4$ cg clone http://www.hawaga.org.uk/ 
gitcompletion.git
defaulting to local storage area
20:46:16 URL:http://www.hawaga.org.uk/gitcompletion.git/HEAD [41/41] - 
 > "refs/heads/.origin-fetching" [1]
progress: 28 objects, 7901 bytes
http://www.hawaga.org.uk/gitcompletion.git/refs/tags/:
20:46:36 ERROR 403: Forbidden.

FINISHED --20:46:36--
Downloaded: 0 bytes in 0 files
New branch: a108bdc110dad770ec5c092759a8bc511790d21f
Cloned to gitcompletion/ (origin http://www.hawaga.org.uk/ 
gitcompletion.git available as branch "origin")


-- 
Ben ベン Бэн
http://www.hawaga.org.uk/ben/

Re: Errors cloning over http -- git-clone and cg-clone fail to fetch a reachable object...

[Petr Baudis]

Dear diary, on Wed, Nov 09, 2005 at 10:49:30AM CET, I got a letter
where Ben Clifford <benc@hawaga.org.uk> said that...
> 
> On 9 Nov 2005, at 12:09, Nick Hengeveld wrote:
> >
> >Those curl result codes all look wrong, and sounds like a memory issue
> >that Johannes Schindelin recently fixed in commit
> >90279074ca5cc336a8bfffd47d19d089b291b432.  Does your git build have  
> >that
> >patch?
> 
> I did not - I've pulled and rebuilt from master, and have that commit  
> now.
> It works better (not perfectly (there's a tags 403) but I suspect  
> that may be permissions config at my server end or absence of tags in  
> my repo or something mumble). I can do this a bunch of times in a row  
> with the same results and the resulting cloned repository looks sane.

This likely means that you don't permit directory listing of the tags/
subdir. Since Cogito is just about to cease using it, though, it's up to
you whether you'll go through the trouble of enabling it or waiting for
new Cogito version. ;-)

-- 
				Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
VI has two modes: the one in which it beeps and the one in which
it doesn't.