Problem cloning the Linux history tree

Problem cloning the Linux history tree

[Marcel Holtmann]

Hi guys,

I have some problems cloning the Linux history tree from Thomas Gleixner
over the HTTP transport. The rsync transport seems to work.

# cg-clone http://www.kernel.org/pub/scm/linux/kernel/git/tglx/history.git
defaulting to local storage area
09:53:02 URL:http://www.kernel.org/pub/scm/linux/kernel/git/tglx/history.git/HEAD [41/41] -> "refs/heads/.origin-fetching" [1]
Getting alternates list
Getting pack list
error: The requested URL returned error: 404
error: Unable to find e7e173af42dbf37b1d946f9ee00219cb3b2bea6a under http://www.kernel.org/pub/scm/linux/kernel/git/tglx/history.git/

Cannot obtain needed object e7e173af42dbf37b1d946f9ee00219cb3b2bea6a
while processing commit 0000000000000000000000000000000000000000.
cg-fetch: objects fetch failed
cg-clone: fetch failed

Regards

Marcel

Re: Problem cloning the Linux history tree

[Junio C Hamano]

Marcel Holtmann <marcel@holtmann.org> writes:

> # cg-clone http://www.kernel.org/pub/scm/linux/kernel/git/tglx/history.git
> defaulting to local storage area
> 09:53:02 URL:http://www.kernel.org/pub/scm/linux/kernel/git/tglx/history.git/HEAD [41/41] -> "refs/heads/.origin-fetching" [1]
> Getting alternates list
> Getting pack list
> error: The requested URL returned error: 404

That repository is packed but lacks objects/info/packs file.
Perhaps you need to ask tglx to run git-update-server-info in
the repository.

Cloning over git native transport should work, though:

    $ git clone git://git.kernel.org/pub/scm/lin.../history.git history

Re: Problem cloning the Linux history tree

[Wolfgang Denk]

In message <7vmzkl3yl0.fsf@assigned-by-dhcp.cox.net> you wrote:
> 
> > # cg-clone http://www.kernel.org/pub/scm/linux/kernel/git/tglx/history.git
> > defaulting to local storage area
> > 09:53:02 URL:http://www.kernel.org/pub/scm/linux/kernel/git/tglx/history.git/HEAD [41/41] -> "refs/heads/.origin-fetching" [1]
...
> That repository is packed but lacks objects/info/packs file.
> Perhaps you need to ask tglx to run git-update-server-info in
> the repository.

I have the same (or a very similar) problem with our Linux repo:

-> cg-clone http://www.denx.de/git/linux-2.6-denx.git foo
...
defaulting to local storage area
09:54:53 URL:http://www.denx.de/git/linux-2.6-denx.git/HEAD [41/41] -> "refs/heads/.origin-fetching" [1]
Getting alternates list
Getting pack list
Getting index for pack eceba3479417b338c2e156b1c6dfd69f3f0eaad1
Getting index for pack b3c6fbdfa36a326815de6358885c7a570a986b1b
Getting index for pack 6122f3ffa476a089122c0bffbbb7a7ce31304b56
Getting index for pack 06949712bb970fc1dd2e13f50accdcd2bf992536
Getting index for pack 459148e2ea863c15479d69ed4886a6f200c79a49
Getting pack 459148e2ea863c15479d69ed4886a6f200c79a49
which contains 010a4c0059a82b834b4d7f65d5cdd4a9717d8ea7
progress: 3 objects, 0 bytes
Getting pack 06949712bb970fc1dd2e13f50accdcd2bf992536
which contains 5014bfa48ac169e0748e1e9651897788feb306dc
progress: 12210 objects, 0 bytes
09:58:42 ERROR 404: Not Found.
09:58:42 URL:http://www.denx.de/git/linux-2.6-denx.git/refs/tags/DENX-2005-10-02-18:30 [41/41] -> "refs/tags/DENX-2005-10-02-18:30" [1]
...
09:58:43 URL:http://www.denx.de/git/linux-2.6-denx.git/refs/tags/v2.6.14-rc5 [41/41] -> "refs/tags/v2.6.14-rc5" [1]
FINISHED --09:58:43--
Downloaded: 19,240 bytes in 30 files
Missing object of tag v2.6.14-rc3... unable to retrieve
New branch: 010a4c0059a82b834b4d7f65d5cdd4a9717d8ea7
Cloned to foo/ (origin http://www.denx.de/git/linux-2.6-denx.git available as branch "origin")

And I did run git-update-server-info, even with --force.


In the server log I see messages like these:

[Fri Nov 04 09:54:52 2005] [error] [client 84.150.96.170] File does not exist: /srv/git/linux-2.6-denx.git/objects/01
[Fri Nov 04 09:54:52 2005] [error] [client 84.150.96.170] File does not exist: /srv/git/linux-2.6-denx.git/objects/info/http-alternates
[Fri Nov 04 09:54:53 2005] [error] [client 84.150.96.170] File does not exist: /srv/git/linux-2.6-denx.git/objects/info/alternates
[Fri Nov 04 09:55:12 2005] [error] [client 84.150.96.170] File does not exist: /srv/git/linux-2.6-denx.git/objects/5b
[Fri Nov 04 09:55:12 2005] [error] [client 84.150.96.170] File does not exist: /srv/git/linux-2.6-denx.git/objects/ca
[Fri Nov 04 09:55:12 2005] [error] [client 84.150.96.170] File does not exist: /srv/git/linux-2.6-denx.git/objects/50
[Fri Nov 04 09:55:12 2005] [error] [client 84.150.96.170] File does not exist: /srv/git/linux-2.6-denx.git/objects/98
[Fri Nov 04 09:55:12 2005] [error] [client 84.150.96.170] File does not exist: /srv/git/linux-2.6-denx.git/objects/79
[Fri Nov 04 09:55:12 2005] [error] [client 84.150.96.170] File does not exist: /srv/git/linux-2.6-denx.git/objects/4e
[Fri Nov 04 09:55:12 2005] [error] [client 84.150.96.170] File does not exist: /srv/git/linux-2.6-denx.git/objects/79
[Fri Nov 04 09:55:12 2005] [error] [client 84.150.96.170] File does not exist: /srv/git/linux-2.6-denx.git/objects/c1
[Fri Nov 04 09:55:12 2005] [error] [client 84.150.96.170] File does not exist: /srv/git/linux-2.6-denx.git/objects/f9
[Fri Nov 04 09:55:12 2005] [error] [client 84.150.96.170] File does not exist: /srv/git/linux-2.6-denx.git/objects/e5
[Fri Nov 04 09:55:13 2005] [error] [client 84.150.96.170] File does not exist: /srv/git/linux-2.6-denx.git/objects/e1
[Fri Nov 04 09:55:13 2005] [error] [client 84.150.96.170] File does not exist: /srv/git/linux-2.6-denx.git/objects/67
[Fri Nov 04 09:55:13 2005] [error] [client 84.150.96.170] File does not exist: /srv/git/linux-2.6-denx.git/objects/0c
[Fri Nov 04 09:55:13 2005] [error] [client 84.150.96.170] File does not exist: /srv/git/linux-2.6-denx.git/objects/b6
[Fri Nov 04 09:55:13 2005] [error] [client 84.150.96.170] File does not exist: /srv/git/linux-2.6-denx.git/objects/bc

These directories really don't exist:

$ cd /srv/git/linux-2.6-denx.git/objects
. ls -l
total 8
drwxrwxr-x  2 wd wd 4096 Nov  4 09:52 info
drwxrwxr-x  2 wd wd 4096 Nov  3 15:15 pack

The question is : why are they requested?

> Cloning over git native transport should work, though:
> 
>     $ git clone git://git.kernel.org/pub/scm/lin.../history.git history

This does not work for me,  either.  It  runs  a  long  time  without
obvious problems, but finally fails with:

-> git clone http://www.denx.de/git/linux-2.6-denx.git bar
...
walk c264852726dde251a0c09ec22f61a9be8b0db68b
walk 69903d6500c73af8329a5fba7153b0d50748981c
walk b59a9504cb93db7fae31e60760725d48652a1fc3
error: Could not interpret tags/DENX-2005-10-02-18:30 as something to pull



I have to admit that I have absolutely no clue (1)  what  might  have
caused this situation nor (2) how to fix it :-(

Best regards,

Wolfgang Denk

-- 
Software Engineering:  Embedded and Realtime Systems,  Embedded Linux
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@denx.de
 The software required `Windows 95 or better', so I installed Linux.

Re: Problem cloning the Linux history tree

[Petr Baudis]

Dear diary, on Fri, Nov 04, 2005 at 10:21:46AM CET, I got a letter
where Wolfgang Denk <wd@denx.de> told me that...
> In message <7vmzkl3yl0.fsf@assigned-by-dhcp.cox.net> you wrote:
> > 
> > > # cg-clone http://www.kernel.org/pub/scm/linux/kernel/git/tglx/history.git
> > > defaulting to local storage area
> > > 09:53:02 URL:http://www.kernel.org/pub/scm/linux/kernel/git/tglx/history.git/HEAD [41/41] -> "refs/heads/.origin-fetching" [1]
> ...
> > That repository is packed but lacks objects/info/packs file.
> > Perhaps you need to ask tglx to run git-update-server-info in
> > the repository.
> 
> I have the same (or a very similar) problem with our Linux repo:
> 
> -> cg-clone http://www.denx.de/git/linux-2.6-denx.git foo
> ...
> defaulting to local storage area
> 09:54:53 URL:http://www.denx.de/git/linux-2.6-denx.git/HEAD [41/41] -> "refs/heads/.origin-fetching" [1]
> Getting alternates list
> Getting pack list
> Getting index for pack eceba3479417b338c2e156b1c6dfd69f3f0eaad1
> Getting index for pack b3c6fbdfa36a326815de6358885c7a570a986b1b
> Getting index for pack 6122f3ffa476a089122c0bffbbb7a7ce31304b56
> Getting index for pack 06949712bb970fc1dd2e13f50accdcd2bf992536
> Getting index for pack 459148e2ea863c15479d69ed4886a6f200c79a49
> Getting pack 459148e2ea863c15479d69ed4886a6f200c79a49
> which contains 010a4c0059a82b834b4d7f65d5cdd4a9717d8ea7
> progress: 3 objects, 0 bytes
> Getting pack 06949712bb970fc1dd2e13f50accdcd2bf992536
> which contains 5014bfa48ac169e0748e1e9651897788feb306dc
> progress: 12210 objects, 0 bytes
> 09:58:42 ERROR 404: Not Found.
> 09:58:42 URL:http://www.denx.de/git/linux-2.6-denx.git/refs/tags/DENX-2005-10-02-18:30 [41/41] -> "refs/tags/DENX-2005-10-02-18:30" [1]
> ...
> 09:58:43 URL:http://www.denx.de/git/linux-2.6-denx.git/refs/tags/v2.6.14-rc5 [41/41] -> "refs/tags/v2.6.14-rc5" [1]
> FINISHED --09:58:43--
> Downloaded: 19,240 bytes in 30 files
> Missing object of tag v2.6.14-rc3... unable to retrieve
> New branch: 010a4c0059a82b834b4d7f65d5cdd4a9717d8ea7
> Cloned to foo/ (origin http://www.denx.de/git/linux-2.6-denx.git available as branch "origin")
> 
> And I did run git-update-server-info, even with --force.

And what exactly is the problem? It seemed to clone ok.

> In the server log I see messages like these:
> 
> [Fri Nov 04 09:54:52 2005] [error] [client 84.150.96.170] File does not exist: /srv/git/linux-2.6-denx.git/objects/01
..snip..
> [Fri Nov 04 09:55:13 2005] [error] [client 84.150.96.170] File does not exist: /srv/git/linux-2.6-denx.git/objects/bc

The fetching code first tries to look into the unpacked object store
(that are those errors) and if that fails, it grabs a pack.

-- 
				Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
VI has two modes: the one in which it beeps and the one in which
it doesn't.

Re: Problem cloning the Linux history tree

[Wolfgang Denk]

In message <20051104092626.GL1431@pasky.or.cz> you wrote:
> 
> > 09:58:42 ERROR 404: Not Found.
> > 09:58:42 URL:http://www.denx.de/git/linux-2.6-denx.git/refs/tags/DENX-2005-10-02-18:30 [41/41] -> "refs/tags/DENX-2005-10-02-18:30" [1]
> > ...
> > Missing object of tag v2.6.14-rc3... unable to retrieve


> And what exactly is the problem? It seemed to clone ok.

There was a 404 error, and the "Missing object of tag" message.



Best regards,

Wolfgang Denk

-- 
Software Engineering:  Embedded and Realtime Systems,  Embedded Linux
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@denx.de
It usually takes more than three weeks to prepare  a  good  impromptu
speech.                                                  - Mark Twain

Re: Problem cloning the Linux history tree

[Junio C Hamano]

Petr Baudis <pasky@suse.cz> writes:

>> 09:58:42 URL:http://www.denx.de/git/linux-2.6-denx.git/refs/tags/DENX-2005-10-02-18:30 [41/41] -> "refs/tags/DENX-2005-10-02-18:30" [1]

I do not know if Cogito works it around or the above just
happens to work, but (notice colon there) tag names that do not
pass git-check-ref-format(1) are not supported.

>> And I did run git-update-server-info, even with --force.

You may want to double check that all your refs appear in
denx.git/info/refs file, and similarly all your packs are
described in denx.git/objects/info/packs file.

> And what exactly is the problem? It seemed to clone ok.

Oh, if that is the case I would stop worrying about it, then.

However, there is this bit that looks suspicious.

>> Missing object of tag v2.6.14-rc3... unable to retrieve

This does not look like an error message I generate, so I am
assuming it is coming from cg-fetch.

Re: Problem cloning the Linux history tree

[Junio C Hamano]

Wolfgang Denk <wd@denx.de> writes:

> Missing object of tag v2.6.14-rc3... unable to retrieve

I do not have a clue on this one; I am assuming that it is
coming from cg-fetch.

> [Fri Nov 04 09:55:13 2005] [error] [client 84.150.96.170] File does not exist: /srv/git/linux-2.6-denx.git/objects/bc
>
> These directories really don't exist:
>
> $ cd /srv/git/linux-2.6-denx.git/objects
> . ls -l
> total 8
> drwxrwxr-x  2 wd wd 4096 Nov  4 09:52 info
> drwxrwxr-x  2 wd wd 4096 Nov  3 15:15 pack

Quick question.  Is the source repository fully packed (i.e. git
repack followed by git prune)?  If that is the case there is
nothing to worry about.  The latest git prunes empty
$GIT_OBJECT_DIRECTORY/??/ directories and recreates them on
demand.

> The question is : why are they requested?

Fetching over http is what we call a "commit walker" -- it
learns the top commit object names, gets that object and reads
it to learn what other objects it needs to slurp, and keeps
asking the next object, one by one.  When dealing with a packed
repository, it first asks individual objects (expecting they
would be found in $GIT_OBJECT_DIRECTORY/??/ directory), and when
not found it fetchs packs and looks inside.  Recent commit
walker knows how to issue multiple requests while earlier
request is still in transit, so you can expect some requests to
individual objects.  That is benign.

>> Cloning over git native transport should work, though:
>> 
>>     $ git clone git://git.kernel.org/pub/scm/lin.../history.git history
>
> This does not work for me,  either.

The above uses git:// URL  but your failure below uses http://

> -> git clone http://www.denx.de/git/linux-2.6-denx.git bar
> ...
> walk c264852726dde251a0c09ec22f61a9be8b0db68b
> walk 69903d6500c73af8329a5fba7153b0d50748981c
> walk b59a9504cb93db7fae31e60760725d48652a1fc3
> error: Could not interpret tags/DENX-2005-10-02-18:30 as something to pull

This one I can answer.  It is the colon in your tag name.
Please rename it to make git-check-ref-format(1) happy, run
git-update-server-info(1) and try again.

Re: Problem cloning the Linux history tree

[Wolfgang Denk]

In message <7vacgkzqau.fsf@assigned-by-dhcp.cox.net> you wrote:
>
> > Missing object of tag v2.6.14-rc3... unable to retrieve
> 
> I do not have a clue on this one; I am assuming that it is
> coming from cg-fetch.
> 


> Quick question.  Is the source repository fully packed (i.e. git
> repack followed by git prune)?  If that is the case there is

Yes, it is. I decided to run repack + update-server-info hoping  that
would help to cleanup the state.

> nothing to worry about.  The latest git prunes empty
> $GIT_OBJECT_DIRECTORY/??/ directories and recreates them on
> demand.

I see. Thanks.

> > error: Could not interpret tags/DENX-2005-10-02-18:30 as something to pull
> 
> This one I can answer.  It is the colon in your tag name.
> Please rename it to make git-check-ref-format(1) happy, run
> git-update-server-info(1) and try again.

Understood. Thanks.

Best regards,

Wolfgang Denk

-- 
Software Engineering:  Embedded and Realtime Systems,  Embedded Linux
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@denx.de
No one can guarantee the actions of another.
	-- Spock, "Day of the Dove", stardate unknown

Re: Problem cloning the Linux history tree

[Marcel Holtmann]

Hi Thomas,

> > # cg-clone http://www.kernel.org/pub/scm/linux/kernel/git/tglx/history.git
> > defaulting to local storage area
> > 09:53:02 URL:http://www.kernel.org/pub/scm/linux/kernel/git/tglx/history.git/HEAD [41/41] -> "refs/heads/.origin-fetching" [1]
> > Getting alternates list
> > Getting pack list
> > error: The requested URL returned error: 404
> 
> That repository is packed but lacks objects/info/packs file.
> Perhaps you need to ask tglx to run git-update-server-info in
> the repository.

can please try to fix it according to Junio's proposal.

Regards

Marcel

Re: Problem cloning the Linux history tree

[Thomas Gleixner]

On Friday 04 November 2005 16:47, Marcel Holtmann wrote:
>
> can please try to fix it according to Junio's proposal.
>

Done
	tglx

Re: Problem cloning the Linux history tree

[Marcel Holtmann]

Hi Thomas,

> > can please try to fix it according to Junio's proposal.
> >
> 
> Done

looks like it is working now. Thanks.

Regards

Marcel