HTTP repo referencing stale heads (can't clone)

HTTP repo referencing stale heads (can't clone)

[Daniel Drake]

Hi,

I maintain a small git repo. I upload it over ssh (with git-push) to a 
machine where it is distributed over http:

http://dsd.object4.net/git/zd1211.git/

For some reason it is no longer possible to clone this repo over http:

walk 35afe6b3a859242a18812e7485ea8b211e24abaf
walk 93d9a9f469282e1e392c16ce571da4c08805e8bb
error: Couldn't get 
http://dsd.object4.net/git/zd1211.git/refs/heads/softmac-old for 
heads/softmac-old
The requested URL returned error: 404
error: Could not interpret heads/softmac-old as something to pull

"softmac-old" is an old branch, which I have recently deleted. I deleted 
it by removing the .git/refs/heads/softmac-old file, and relying on 
git-prune to clear out old objects.

Even on the server-side, there is no obvious reference to this old head:

$ find -name '*softmac*'
$ grep -R softmac *
(no results for either)

"git-fsck-objects" reports nothing, "git-fsck-objects --full" reports:
dangling commit 7cc423c942975005f96f308186537ad6e7808c2e
dangling commit b36378de6231f1b5100b1517b9c8c243a21090fd

I have tried running git-prune and git-update-server-info, but that 
doesn't help.

Any ideas? I'm still new to git.
I am running git-1.2.4

Thanks,
Daniel

Re: HTTP repo referencing stale heads (can't clone)

[Junio C Hamano]

Daniel Drake <dsd@gentoo.org> writes:

> I maintain a small git repo. I upload it over ssh (with git-push) to a
> machine where it is distributed over http:
>
> http://dsd.object4.net/git/zd1211.git/
>...
> I have tried running git-prune and git-update-server-info, but that
> doesn't help.
>
> Any ideas?

My standard answer would be

http-server$ cd /var/www/git/zd1211.git/ ;# or whereever
http-server$ GIT_DIR=. git-update-server-info

but you said you have run it already...  Can you try to see if
there is some caching proxy involved that still serves stale
info/refs file?

client$ wget http://dsd.object4.net/git/zd1211.git/info/refs

Re: HTTP repo referencing stale heads (can't clone)

[Nick Hengeveld]

On Mon, Apr 03, 2006 at 10:23:22AM -0700, Junio C Hamano wrote:

> My standard answer would be
> 
> http-server$ cd /var/www/git/zd1211.git/ ;# or whereever
> http-server$ GIT_DIR=. git-update-server-info

Is there any interest in making the HTTP transport slighly less dumb by
using DAV?

I have a working patch to http-fetch that tries to use PROPFIND to get a
remote pack list and falls back to using objects/info/packs.  It's
feasible to do something similar to get a remote ref list when cloning,
although that's a bit more work as all refs would have to be fetched
into a local repo and parsed to determine the object type.

Long term, this could give a repo admin the choice of either making sure
git-update-server-info has been run after every ref/pack change or
enabling DAV once.  Assuming they need to use HTTP.

-- 
For a successful technology, reality must take precedence over public
relations, for nature cannot be fooled.

Re: HTTP repo referencing stale heads (can't clone)

[Daniel Drake]

Junio C Hamano wrote:
> client$ wget http://dsd.object4.net/git/zd1211.git/info/refs

Ah, should have known. I am behind a (lame) transparent proxy on port 80.

I opened that file in my web browser and it showed the old heads. After 
a force-refresh (ctrl+F5, which sends some additionally http headers to 
refresh the page from the real server), the old heads disappeared, and 
git now clones successfully.

git-http-fetch should probably send those extra headers too. I'll try to 
find some time to look at this next week.

Thanks!
Daniel

Re: HTTP repo referencing stale heads (can't clone)

[Junio C Hamano]

Nick Hengeveld <nickh@reactrix.com> writes:

> Is there any interest in making the HTTP transport slighly less dumb by
> using DAV?

I personally feel PROPFIND is the right way to do "wget -r", and
very much welcome a patch to replace objects/info/packs with it
when able.

> I have a working patch to http-fetch that tries to use PROPFIND to get a
> remote pack list and falls back to using objects/info/packs.  It's
> feasible to do something similar to get a remote ref list when cloning,
> although that's a bit more work as all refs would have to be fetched
> into a local repo and parsed to determine the object type.

Faking info/refs with PROPFIND, if we do not have to peel the
onion ^{}, should be relatively cheap operation, and could be
done as an enhancement to git-ls-remote.sh.  If your faked
info/refs file lacks ^{} entries, git-fetch cannot auto-follow
tags, but git-clone should work as before.

A clever sysadmins could mod_rewrite requests to info/refs and
objects/info/packs with a custom CGI, but then probably they
would be running git-daemon ;-).

Re: HTTP repo referencing stale heads (can't clone)

[Petr Baudis]

Dear diary, on Mon, Apr 03, 2006 at 08:09:30PM CEST, I got a letter
where Nick Hengeveld <nickh@reactrix.com> said that...
> Long term, this could give a repo admin the choice of either making sure
> git-update-server-info has been run after every ref/pack change or
> enabling DAV once.  Assuming they need to use HTTP.

Well, what is the actual advantage of DAV compared to
git-update-server-info? Why would I prefer enabling DAV?

-- 
				Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
Right now I am having amnesia and deja-vu at the same time.  I think
I have forgotten this before.

Re: HTTP repo referencing stale heads (can't clone)

[Nick Hengeveld]

On Tue, Apr 04, 2006 at 12:00:35PM +0200, Petr Baudis wrote:

> Well, what is the actual advantage of DAV compared to
> git-update-server-info? Why would I prefer enabling DAV?

In theory, things should work the same either way.  It seems that in
practice though, the server info files continue to surface as a source
of fetch problems.

-- 
For a successful technology, reality must take precedence over public
relations, for nature cannot be fooled.

Re: HTTP repo referencing stale heads (can't clone)

[Petr Baudis]

Dear diary, on Tue, Apr 04, 2006 at 02:10:57PM CEST, I got a letter
where Nick Hengeveld <nickh@reactrix.com> said that...
> On Tue, Apr 04, 2006 at 12:00:35PM +0200, Petr Baudis wrote:
> 
> > Well, what is the actual advantage of DAV compared to
> > git-update-server-info? Why would I prefer enabling DAV?
> 
> In theory, things should work the same either way.  It seems that in
> practice though, the server info files continue to surface as a source
> of fetch problems.

Because people do not know they have to set up their post-update hook.
When they are already going at lengths to find out how to set up DAV for
git fetch, they would discover the post-update hook way as well.

So, it really seems rather redundant to me.

-- 
				Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
Right now I am having amnesia and deja-vu at the same time.  I think
I have forgotten this before.

Re: HTTP repo referencing stale heads (can't clone)

[Nick Hengeveld]

On Tue, Apr 04, 2006 at 05:27:37PM +0200, Petr Baudis wrote:

> Because people do not know they have to set up their post-update hook.
> When they are already going at lengths to find out how to set up DAV for
> git fetch, they would discover the post-update hook way as well.

There are other reasons that a client can end up with a stale copy of
the server info files - in this case the problem was an intermediate
proxy out of the control of the repo admin.

While we should be able to fix that particular problem, it seems safer
to go straight to the source if possible.

> So, it really seems rather redundant to me.

If I've already enabled DAV for pushing to a repo, I'd find it nice to
be able to use it for fetches as well.

-- 
For a successful technology, reality must take precedence over public
relations, for nature cannot be fooled.

Re: HTTP repo referencing stale heads (can't clone)

[Nick Hengeveld]

On Mon, Apr 03, 2006 at 07:28:28PM +0100, Daniel Drake wrote:

> Ah, should have known. I am behind a (lame) transparent proxy on port 80.
> 
> I opened that file in my web browser and it showed the old heads. After 
> a force-refresh (ctrl+F5, which sends some additionally http headers to 
> refresh the page from the real server), the old heads disappeared, and 
> git now clones successfully.
> 
> git-http-fetch should probably send those extra headers too. I'll try to 
> find some time to look at this next week.

git-http-fetch uses the "Pragma: no-cache" header when requesting
objects that shouldn't be cached.  Is this the additional header you're
referring to?

This patch adds the header to git-ls-remote for the info/refs request.


git-ls-remote: send no-cache header when fetching info/refs

Proxies should not cache this file as it can cause a client to end up with
a stale version, as reported here:

http://marc.theaimsgroup.com/?l=git&m=114407944125389

Signed-off-by: Nick Hengeveld <nickh@reactrix.com>


---

 git-ls-remote.sh |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

da9b6fa01f1a8bd6ab5f6d4346584f3f032584aa
diff --git a/git-ls-remote.sh b/git-ls-remote.sh
index 2c9a588..b6882a9 100755
--- a/git-ls-remote.sh
+++ b/git-ls-remote.sh
@@ -53,7 +53,7 @@ http://* | https://* )
         if [ -n "$GIT_SSL_NO_VERIFY" ]; then
             curl_extra_args="-k"
         fi
-	curl -nsf $curl_extra_args "$peek_repo/info/refs" ||
+	curl -nsf $curl_extra_args --header "Pragma: no-cache" "$peek_repo/info/refs" ||
 		echo "failed	slurping"
 	;;
 
-- 
1.3.0.rc1.g9aef-dirty

Re: HTTP repo referencing stale heads (can't clone)

[Radoslaw Szkodzinski]

[-- Attachment #1: Type: text/plain, Size: 988 bytes --]

Nick Hengeveld wrote:
> On Mon, Apr 03, 2006 at 07:28:28PM +0100, Daniel Drake wrote:
> 
>> Ah, should have known. I am behind a (lame) transparent proxy on port 80.
>>
>> I opened that file in my web browser and it showed the old heads. After 
>> a force-refresh (ctrl+F5, which sends some additionally http headers to 
>> refresh the page from the real server), the old heads disappeared, and 
>> git now clones successfully.
>>
>> git-http-fetch should probably send those extra headers too. I'll try to 
>> find some time to look at this next week.
> 
> git-http-fetch uses the "Pragma: no-cache" header when requesting
> objects that shouldn't be cached.  Is this the additional header you're
> referring to?
> 

As per HTTP 1.1, it should also send:
Cache-Control: no-cache

Pragma: no-cache is the deprecated extension.
It's safe to send both.

-- 
GPG Key id:  0xD1F10BA2
Fingerprint: 96E2 304A B9C4 949A 10A0  9105 9543 0453 D1F1 0BA2

AstralStorm


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 254 bytes --]