Corruption: empty refs/heads in otherwise filled repo: cannot clone?

Corruption: empty refs/heads in otherwise filled repo: cannot clone?

[Jan Wielemaker]

Hi,

I'm a bit puzzled. I have a bare repository, somehow without any files
in refs/heads.

It started using conversion from CVS via SVN using git-1.5.3.4. It is
configured for group access and there are a number of SSH accounts on
the machine in the proper group using git-shell for fellow developers. 

I never have any problems, but after upgrading to git-git (1.5.6.rc3)
fellow users get error:

error: no such remote ref refs/heads/JPL31
error: no such remote ref refs/heads/V57X
error: no such remote ref refs/heads/V5_4_patches
error: no such remote ref refs/heads/XML_UNICODE
error: no such remote ref refs/heads/attvar
error: no such remote ref refs/heads/gmp

Creating a new account and doing a fresh clone I even got the fatal
error there are no remote branches!? Still, using my own user id, I
could do anything I wanted. I checked permissions to see whether there
are differences between group and owner permissions, but couldn't find
anything suspicious.  git fsck --full on the repo gives no errors.

It turns out the directory refs/heads is empty!?

I made a fresh clone of the repo as myself, creating all references
nicely in refs/remotes/origin. Then I copied these files to refs/heads
into the main bare repository and now all appears to work nicely again.
Still, this is a bit worrying ...  My questions:

    * Is my work-around safe and sound?
    * How can this have happened?
    * Why can I clone as myself and not as anyone else!?  Stranger:
    after a "sudo -u <someone> /bin/bash" I could not clone.  I can
    copy the entire tree using cp -a, but I still cannot clone the
    copy with this user, while as myself I can clone the copy. 
    Somehow `as myself' appears to get the branchheads from somewhere.
    There is only one copy of git installed, so both users use the
    same.  What can cause this difference?

	Thanks --- Jan

Re: Corruption: empty refs/heads in otherwise filled repo: cannot clone?

[Jakub Narebski]

Jan Wielemaker <J.Wielemaker@uva.nl> writes:

> I'm a bit puzzled. I have a bare repository, somehow without any files
> in refs/heads.

Do you have .git/packed-refs file? How do you have gc.packrefs set
("git config --get gc.packrefs")? What are the git version on the
clients that have trouble accessing repository, and what protocol
(transport merhod) do they use: "file", http(s)/ftp, git, ssh?

-- 
Jakub Narebski
Poland
ShadeHawk on #git

Re: Corruption: empty refs/heads in otherwise filled repo: cannot clone?

[Jan Wielemaker]

Hi Jakub,

Thanks for the quick reply.  Info:

On Monday 30 June 2008 12:10, Jakub Narebski wrote:
> Jan Wielemaker <J.Wielemaker@uva.nl> writes:
> > I'm a bit puzzled. I have a bare repository, somehow without any files
> > in refs/heads.
>
> Do you have .git/packed-refs file? How do you have gc.packrefs set

Its a bare repo, so I'll forget the .git.  Yes, I have that file and
it contains nice references,  I checked a few by hand, and they have
the same SHA1 as the files I copied.

> ("git config --get gc.packrefs")? What are the git version on the

This gives no output, so I guess the answer is 'no'.  Is that wrong?
This is indeed the case (config):

----------------------------------------------------------------
[core]
        repositoryformatversion = 0
        filemode = true
        bare = true
        sharedrepository = 1
[gitcvs]
        enabled = 1
        logfile = /home/git/pl.git/cvs.log
        allbinary = 1
----------------------------------------------------------------

> clients that have trouble accessing repository, and what protocol
> (transport merhod) do they use: "file", http(s)/ftp, git, ssh?

I could reproduce the problem using the same git that maintains the bare
repository, which started as git-1.5.3.4 and is now 1.5.6.rc3. It
reproduces both using local file access and ssh. Other people have a
variety of versions. I know of concrete problems using the stable 1.5.5
over ssh.

The problem that I can't even clone is indeed after running a git-gc.

So, now I have all branch head refs double (in packed-refs and in
refs/heads). That explains ar least where they come from, but it fails
to explain why doing exactly the same using the same git on the same
repo as two users yields a different result. Of course, besides the
desire to understand how his works, I'm mostly interested in how to fix
this :-)

	Thanks --- Jan

Re: Corruption: empty refs/heads in otherwise filled repo: cannot clone?

[Jakub Narebski]

Jan Wielemaker wrote:
> On Monday 30 June 2008 12:10, Jakub Narebski wrote:
>> Jan Wielemaker <J.Wielemaker@uva.nl> writes:
>>>
>>> I'm a bit puzzled. I have a bare repository, somehow without any files
>>> in refs/heads.
>>
>> Do you have .git/packed-refs file?
> 
> Its a bare repo, so I'll forget the .git.  Yes, I have that file and
> it contains nice references,  I checked a few by hand, and they have
> the same SHA1 as the files I copied.

That is where the refs are instead of being in individual files under
refs/ - those are so called "packed refs".

  When a ref is  missing from  the traditional $GIT_DIR/refs hierarchy,
  it is looked up in $GIT_DIR/packed-refs and used if found.

I'm not sure why branches are also packed, because git used to pack
only tags and refs which were packed already.

>> How do you have gc.packrefs set ("git config --get gc.packrefs")?
> 
> This gives no output, so I guess the answer is 'no'.

In git-config(1) you can find the following:

  gc.packrefs::
        `git gc` does not run `git pack-refs` in a bare repository by
        default so that older dumb-transport clients can still fetch
        from the repository.  Setting this to `true` lets `git
        gc` to run `git pack-refs`.  Setting this to `false` tells
        `git gc` never to run `git pack-refs`. The default setting is
        `notbare`. Enable it only when you know you do not have to
        support such clients.  The default setting will change to `true`
        at some stage, and setting this to `false` will continue to
        prevent `git pack-refs` from being run from `git gc`.

Unless something changed (and git Documentation was not updated) git
should not pack refs by default.

>> What are the git version on the
>> clients that have trouble accessing repository, and what protocol
>> (transport method) do they use: "file", http(s)/ftp, git, ssh?
> 
> I could reproduce the problem using the same git that maintains the bare
> repository, which started as git-1.5.3.4 and is now 1.5.6.rc3. It
> reproduces both using local file access and ssh. Other people have a
> variety of versions. I know of concrete problems using the stable 1.5.5
> over ssh.
>
> The problem that I can't even clone is indeed after running a git-gc.

Hmmm... strange. I know that there can be problems with older clients
(those which do not understand packed-refs format) accessing
repositories with packed refs (without loose refs) via "dumb"
protocols.  Problems with accessing repositories locally and via ssh
might point to some troubles with permissions and ownership of
$GIT_DIR and $GIT_DIR/packed-refs.
 
git-gc can pack refs, that is what running it causes problems.

> So, now I have all branch head refs double (in packed-refs and in
> refs/heads). That explains ar least where they come from, but it fails
> to explain why doing exactly the same using the same git on the same
> repo as two users yields a different result. Of course, besides the
> desire to understand how his works, I'm mostly interested in how to fix
> this :-)

Unfortunately as it is not caused by the issue I thought about I cannot
help you further...

-- 
Jakub Narebski
Poland

Re: Corruption: empty refs/heads in otherwise filled repo: cannot clone?

[Jan Wielemaker]

Hi Jakub,

Summarising, I think the conclusion is that git pack-refs has somehow
been run on this repository, and being a bare one this is not a
particulary good idea right now. I have the impression I should `unpack'
them by putting the appriate files in heads (done) and tags (now still)
and (re)move packed-refs.

	Cheers --- Jan

On Monday 30 June 2008 13:26, Jakub Narebski wrote:
> Jan Wielemaker wrote:
> > On Monday 30 June 2008 12:10, Jakub Narebski wrote:
> >> Jan Wielemaker <J.Wielemaker@uva.nl> writes:
> >>> I'm a bit puzzled. I have a bare repository, somehow without any files
> >>> in refs/heads.
> >>
> >> Do you have .git/packed-refs file?
> >
> > Its a bare repo, so I'll forget the .git.  Yes, I have that file and
> > it contains nice references,  I checked a few by hand, and they have
> > the same SHA1 as the files I copied.
>
> That is where the refs are instead of being in individual files under
> refs/ - those are so called "packed refs".
>
>   When a ref is  missing from  the traditional $GIT_DIR/refs hierarchy,
>   it is looked up in $GIT_DIR/packed-refs and used if found.
>
> I'm not sure why branches are also packed, because git used to pack
> only tags and refs which were packed already.
>
> >> How do you have gc.packrefs set ("git config --get gc.packrefs")?
> >
> > This gives no output, so I guess the answer is 'no'.
>
> In git-config(1) you can find the following:
>
>   gc.packrefs::
>         `git gc` does not run `git pack-refs` in a bare repository by
>         default so that older dumb-transport clients can still fetch
>         from the repository.  Setting this to `true` lets `git
>         gc` to run `git pack-refs`.  Setting this to `false` tells
>         `git gc` never to run `git pack-refs`. The default setting is
>         `notbare`. Enable it only when you know you do not have to
>         support such clients.  The default setting will change to `true`
>         at some stage, and setting this to `false` will continue to
>         prevent `git pack-refs` from being run from `git gc`.
>
> Unless something changed (and git Documentation was not updated) git
> should not pack refs by default.
>
> >> What are the git version on the
> >> clients that have trouble accessing repository, and what protocol
> >> (transport method) do they use: "file", http(s)/ftp, git, ssh?
> >
> > I could reproduce the problem using the same git that maintains the bare
> > repository, which started as git-1.5.3.4 and is now 1.5.6.rc3. It
> > reproduces both using local file access and ssh. Other people have a
> > variety of versions. I know of concrete problems using the stable 1.5.5
> > over ssh.
> >
> > The problem that I can't even clone is indeed after running a git-gc.
>
> Hmmm... strange. I know that there can be problems with older clients
> (those which do not understand packed-refs format) accessing
> repositories with packed refs (without loose refs) via "dumb"
> protocols.  Problems with accessing repositories locally and via ssh
> might point to some troubles with permissions and ownership of
> $GIT_DIR and $GIT_DIR/packed-refs.
>
> git-gc can pack refs, that is what running it causes problems.
>
> > So, now I have all branch head refs double (in packed-refs and in
> > refs/heads). That explains ar least where they come from, but it fails
> > to explain why doing exactly the same using the same git on the same
> > repo as two users yields a different result. Of course, besides the
> > desire to understand how his works, I'm mostly interested in how to fix
> > this :-)
>
> Unfortunately as it is not caused by the issue I thought about I cannot
> help you further...

Re: Corruption: empty refs/heads in otherwise filled repo: cannot clone?

[Jakub Narebski]

Jan Wielemaker wrote:

> Summarising, I think the conclusion is that git pack-refs has somehow
> been run on this repository, and being a bare one this is not a
> particulary good idea right now. I have the impression I should `unpack'
> them by putting the appriate files in heads (done) and tags (now still)
> and (re)move packed-refs.

If you use new enough git both on server and on clients it should
not have problems with packed-refs. I would rather check permissions
of $GIT_DIR and $GIT_DIR/packed-refs.

If "git show-ref" and "git for-each-ref" works, then 
"git ls-remote <repo>" should work, and git-fetch/git-clone
shoulw work too...

-- 
Jakub Narebski
Poland

Re: Corruption: empty refs/heads in otherwise filled repo: cannot clone?

[Jan Wielemaker]

On Monday 30 June 2008 14:03, Jakub Narebski wrote:
> Jan Wielemaker wrote:
> > Summarising, I think the conclusion is that git pack-refs has somehow
> > been run on this repository, and being a bare one this is not a
> > particulary good idea right now. I have the impression I should `unpack'
> > them by putting the appriate files in heads (done) and tags (now still)
> > and (re)move packed-refs.
>
> If you use new enough git both on server and on clients it should
> not have problems with packed-refs. I would rather check permissions
> of $GIT_DIR and $GIT_DIR/packed-refs.

There is no permission problem, as a I proved by doing a recursive copy
of the whole repo (cp -a, no errors) and the problem prevails on the
copy. A serious scan for permission errors was my first step. Almost
looks like something in the environment, but I can't find anything weird
there either.

> If "git show-ref" and "git for-each-ref" works, then
> "git ls-remote <repo>" should work, and git-fetch/git-clone
> shoulw work too...

Thanks for the pointers. I'll leave it for now (busy ...). I now
understand how it is supposed to work. If I can find time at some point
I'll run strace in different setups to see whether I can spot the
difference.

Anyway, thanks to your help I am fairly confident my repo is in good
shape again.

	Cheers --- Jan

Re: Corruption: empty refs/heads in otherwise filled repo: cannot clone?

[Daniel Barkalow]

On Mon, 30 Jun 2008, Jan Wielemaker wrote:

> On Monday 30 June 2008 14:03, Jakub Narebski wrote:
> > Jan Wielemaker wrote:
> > > Summarising, I think the conclusion is that git pack-refs has somehow
> > > been run on this repository, and being a bare one this is not a
> > > particulary good idea right now. I have the impression I should `unpack'
> > > them by putting the appriate files in heads (done) and tags (now still)
> > > and (re)move packed-refs.
> >
> > If you use new enough git both on server and on clients it should
> > not have problems with packed-refs. I would rather check permissions
> > of $GIT_DIR and $GIT_DIR/packed-refs.
> 
> There is no permission problem, as a I proved by doing a recursive copy
> of the whole repo (cp -a, no errors) and the problem prevails on the
> copy. A serious scan for permission errors was my first step. Almost
> looks like something in the environment, but I can't find anything weird
> there either.

That's a "cp -a" as somebody else, I assume? (If it were as you or root, 
you'd generate a copy of the repository with any permission problem 
unchanged, since -a includes -p.)

	-Daniel
*This .sig left intentionally blank*

Re: Corruption: empty refs/heads in otherwise filled repo: cannot clone?

[Jan Wielemaker]

On Monday 30 June 2008 21:33:50 Daniel Barkalow wrote:
> On Mon, 30 Jun 2008, Jan Wielemaker wrote:
> > On Monday 30 June 2008 14:03, Jakub Narebski wrote:
> > > Jan Wielemaker wrote:
> > > > Summarising, I think the conclusion is that git pack-refs has somehow
> > > > been run on this repository, and being a bare one this is not a
> > > > particulary good idea right now. I have the impression I should
> > > > `unpack' them by putting the appriate files in heads (done) and tags
> > > > (now still) and (re)move packed-refs.
> > >
> > > If you use new enough git both on server and on clients it should
> > > not have problems with packed-refs. I would rather check permissions
> > > of $GIT_DIR and $GIT_DIR/packed-refs.
> >
> > There is no permission problem, as a I proved by doing a recursive copy
> > of the whole repo (cp -a, no errors) and the problem prevails on the
> > copy. A serious scan for permission errors was my first step. Almost
> > looks like something in the environment, but I can't find anything weird
> > there either.
>
> That's a "cp -a" as somebody else, I assume? (If it were as you or root,
> you'd generate a copy of the repository with any permission problem
> unchanged, since -a includes -p.)

I'm an old time Unix guy :-).  The failing user has a git-shell.  I did
(reconstructing from memory):

	% sudo -u failinguser /bin/bash
	<passwd>
	% Use whoami to validate I was this user
	% git clone /home/git/pl.git
	<failed (no remote branches)>
	% cp -a /home/git/pl.git tmp.git
	<ok, no errors>
	% git clone tmp.git
	<failed (no remote branches)>

Funny enough, as myself I can clone tmp.git!?  I have only one git 
installation
in /usr/local/bin.

	Cheers --- Jan

Re: Corruption: empty refs/heads in otherwise filled repo: cannot clone?

[Björn Steinbrink]

On 2008.06.30 13:26:10 +0200, Jakub Narebski wrote:
> Jan Wielemaker wrote:
> > On Monday 30 June 2008 12:10, Jakub Narebski wrote:
> >> Jan Wielemaker <J.Wielemaker@uva.nl> writes:
> >>>
> >>> I'm a bit puzzled. I have a bare repository, somehow without any files
> >>> in refs/heads.
> >>
> >> Do you have .git/packed-refs file?
> > 
> > Its a bare repo, so I'll forget the .git.  Yes, I have that file and
> > it contains nice references,  I checked a few by hand, and they have
> > the same SHA1 as the files I copied.
> 
> That is where the refs are instead of being in individual files under
> refs/ - those are so called "packed refs".
> 
>   When a ref is  missing from  the traditional $GIT_DIR/refs hierarchy,
>   it is looked up in $GIT_DIR/packed-refs and used if found.
> 
> I'm not sure why branches are also packed, because git used to pack
> only tags and refs which were packed already.
> 
> >> How do you have gc.packrefs set ("git config --get gc.packrefs")?
> > 
> > This gives no output, so I guess the answer is 'no'.
> 
> In git-config(1) you can find the following:
> 
>   gc.packrefs::
>         `git gc` does not run `git pack-refs` in a bare repository by
>         default so that older dumb-transport clients can still fetch
>         from the repository.  Setting this to `true` lets `git
>         gc` to run `git pack-refs`.  Setting this to `false` tells
>         `git gc` never to run `git pack-refs`. The default setting is
>         `notbare`. Enable it only when you know you do not have to
>         support such clients.  The default setting will change to `true`
>         at some stage, and setting this to `false` will continue to
>         prevent `git pack-refs` from being run from `git gc`.
> 
> Unless something changed (and git Documentation was not updated) git
> should not pack refs by default.

I had a quick glance over the log for builtin-gc.c, and that changed
last year in:

56752391 Make "git gc" pack all refs by default

Björn - who lacks time to provide a doc update patch :-(