Tags, Grafts, and Clones, oh my!

Tags, Grafts, and Clones, oh my!

[Stephen Bash]

Hi all-

I'm currently working on migrating my company's SVN repository to git.  Based on a conversation with Ram at the beginning of the summer, I'm using svn-fe plus a couple of my own scripts.  To create the git branches and tags from the svn-fe generated repo I clone a bunch of "mini-repos", which I subdirectory-filter, then git fetch the branches/tags from the mini-repos into a "fusion" repo where I graft everything back together, and finally one last filter-branch to permanently commit the grafts.

Unfortunately I'm running into a problem with cloning the resulting repository.  Any git tags that are not associated with a live branch are declared invalid:

   error: refs/tags/tagFoo does not point to a valid object!

I've now reproduced this issue in micro (much easier to work with than the 20k commits in the real repo), and it does go away if I git checkout -b branchFoo tagFoo before cloning the repository.  I've examined the source repository, and the tag appears valid to me, as does the commit it points to.  The first error I see is during the clone.

Does this situation make sense to anyone?  If it's a potential bug (rather than user error), I can submit my testcase.  I'm currently working with git 1.7.2.1 on MacOS 10.6.4.

Any ideas will be greatly appreciated!

Thanks,
Stephen

Re: Tags, Grafts, and Clones, oh my!

[Ramkumar Ramachandra]

Hi Stephen,

Stephen Bash writes:
> Unfortunately I'm running into a problem with cloning the resulting repository.  Any git tags that are not associated with a live branch are declared invalid:
> 
>    error: refs/tags/tagFoo does not point to a valid object!
> 
> I've now reproduced this issue in micro (much easier to work with than the 20k commits in the real repo), and it does go away if I git checkout -b branchFoo tagFoo before cloning the repository.  I've examined the source repository, and the tag appears valid to me, as does the commit it points to.  The first error I see is during the clone.
> 
> Does this situation make sense to anyone?  If it's a potential bug (rather than user error), I can submit my testcase.  I'm currently working with git 1.7.2.1 on MacOS 10.6.4.

It seems too vague to me. Can you submit your testcase? Perhaps we can
work from there?

-- Ram

Re: Tags, Grafts, and Clones, oh my!

[Stephen Bash]

> >
> >    error: refs/tags/tagFoo does not point to a valid object!
> >
> 
> It seems too vague to me. Can you submit your testcase? Perhaps we can
> work from there?

Sure:

mkdir gitTagBug
cd gitTagBug

git init foo
cd foo
echo A >> foo.txt
git add foo.txt
git commit -m "Created foo"
GRAFT_PARENT=`git rev-parse master`
echo B >> foo.txt
git commit -am "Added B"
cd ..

git init bar
cd bar
echo A >> foo.txt
echo C >> foo.txt
git add foo.txt
git commit -m "Added C"
GRAFT_CHILD=`git rev-parse master`
git tag -am "Tagging foo C" tagFoo $GRAFT_CHILD
cd ..

cd foo
git fetch ../bar refs/tags/tagFoo:refs/tags/tagFoo
echo "$GRAFT_CHILD $GRAFT_PARENT" >> .git/info/grafts
git filter-branch --tag-name-filter cat tagFoo
cd ..

# Purposely use a file URL to prune any unreferenced objects
git clone file:///`pwd`/foo newFoo

My output from the clone:

   Cloning into newFoo...
   remote: Counting objects: 9, done.
   remote: Compressing objects: 100% (3/3), done.
   remote: Total 9 (delta 0), reused 0 (delta 0)
   Receiving objects: 100% (9/9), done.
   error: refs/tags/tagFoo does not point to a valid object!

Thanks,
Stephen

Re: Tags, Grafts, and Clones, oh my!

[Ramkumar Ramachandra]

Hi Stephen,

Stephen Bash writes:
> Sure:
> 
> mkdir gitTagBug
> cd gitTagBug
> 
> git init foo
> cd foo
> echo A >> foo.txt
> git add foo.txt
> git commit -m "Created foo"
> GRAFT_PARENT=`git rev-parse master`
> echo B >> foo.txt
> git commit -am "Added B"
> cd ..
> 
> git init bar
> cd bar
> echo A >> foo.txt
> echo C >> foo.txt
> git add foo.txt
> git commit -m "Added C"
> GRAFT_CHILD=`git rev-parse master`
> git tag -am "Tagging foo C" tagFoo $GRAFT_CHILD
> cd ..
> 
> cd foo
> git fetch ../bar refs/tags/tagFoo:refs/tags/tagFoo
> echo "$GRAFT_CHILD $GRAFT_PARENT" >> .git/info/grafts
> git filter-branch --tag-name-filter cat tagFoo
> cd ..
> 
> # Purposely use a file URL to prune any unreferenced objects
> git clone file:///`pwd`/foo newFoo
> 
> My output from the clone:
> 
>    Cloning into newFoo...
>    remote: Counting objects: 9, done.
>    remote: Compressing objects: 100% (3/3), done.
>    remote: Total 9 (delta 0), reused 0 (delta 0)
>    Receiving objects: 100% (9/9), done.
>    error: refs/tags/tagFoo does not point to a valid object!

Thanks for the testcase! Offhand, it definitely looks like a bug. I'm
investigating to figure out which part of the chain is at fault.

-- Ram

Re: Tags, Grafts, and Clones, oh my!

[Stephen Bash]

> Thanks for the testcase! Offhand, it definitely looks like a bug. I'm
> investigating to figure out which part of the chain is at fault.

No problem.  I was very happily to isolate it outside the large repo I was working with...

Contrary to my first e-mail, 'git checkout -b branchFoo tagFoo' doesn't solve the problem...  My guess is I got lazy and did a "normal" local clone (no file URL) when I was testing.

Data from further testing:
  - doing a "normal" local clone doesn't emit the error
  - a remote clone over ssh does emit the error (so it's not just file:///)
  - in a brand new repo (init'ed, not cloned) 'git fetch ../foo refs/tags/tagFoo:refs/tags/tagFoo' fails:
      error: unable to find 28fffee... (sha of tag object)
  - in a brand new repo 'git fetch ../foo refs/heads/branchFoo:ref/heads/branchFoo' succeeds, and correctly fetches tagFoo  (where branchFoo is created via 'git checkout -b branchFoo tagFoo')


Thanks,
Stephen

refs/original breaks git-clone for tags (was Re: Tags, Grafts, and Clones, oh my!)

[Stephen Bash]

> > Thanks for the testcase! Offhand, it definitely looks like a bug.
> > I'm
> > investigating to figure out which part of the chain is at fault.
> 
> No problem. I was very happily to isolate it outside the large repo I
> was working with...
> 
> Data from further testing:
> - doing a "normal" local clone doesn't emit the error
> - a remote clone over ssh does emit the error (so it's not just
> file:///)
> - in a brand new repo (init'ed, not cloned) 'git fetch ../foo
> refs/tags/tagFoo:refs/tags/tagFoo' fails:
> error: unable to find 28fffee... (sha of tag object)
> - in a brand new repo 'git fetch ../foo
> refs/heads/branchFoo:ref/heads/branchFoo' succeeds, and correctly
> fetches tagFoo (where branchFoo is created via 'git checkout -b
> branchFoo tagFoo')

After a lot of guess and check, it appears the issue is somehow related to the refs/original directory created by filter-branch.  If that directory is moved out of refs/ or deleted the clone succeeds.  Digging further, a simple rename of refs/original/refs/tags/tagFoo to anything else also fixes the problem.

A simplified test case is:

git init foo
cd foo
echo A >> foo.txt
git add foo.txt
git commit -m "Created foo"
git tag -am "Tagging foo" tagFoo
git filter-branch --env-filter 'export GIT_AUTHOR_NAME=xyz123' --tag-name-filter cat -- --all
cd ..
git clone file:///`pwd`/foo newFoo

git clone will "succeed" (exit 0), but throw the error 

   error: refs/tags/tagFoo does not point to a valid object!

and the tagFoo will not exist in the new repo.

(The env-filter is arbitrary, just need something that will force a commit rewrite)  For this bug to occur, the filter-branch must create refs/original/refs/tags/tagFoo, so if the filter-branch command is 

git filter-branch --env-filter 'export GIT_AUTHOR_NAME=xyz123' --tag-name-filter cat master

filter-branch will happily rewrite the tag, but won't create the offending file, so the clone will succeed without error (and the tag will exist in the new repo).  

Removing refs/original is a pretty trivial work-around, so I'm going to modify my scripts and continue working on my SVN transition.  Let me know if I can be of any assistance tracking down the actual bug.

Thanks,
Stephen

Re: refs/original breaks git-clone for tags

[Jonathan Nieder]

Hi Stephen,

Stephen Bash wrote:

> After a lot of guess and check, it appears the issue is somehow
> related to the refs/original directory created by filter-branch.  If
> that directory is moved out of refs/ or deleted the clone succeeds.
> Digging further, a simple rename of refs/original/refs/tags/tagFoo
> to anything else also fixes the problem.

Thanks for a clear report.  I backported your test case:

-- 8< --
#!/bin/sh
mkdir foo &&
(
	cd foo &&
	git init &&
	echo A >foo.txt &&
	git add foo.txt &&
	git commit -m "Created foo" &&
	git tag -am "Tagging foo" tagFoo &&
	git filter-branch --env-filter 'export GIT_AUTHOR_NAME=xyz123' \
				--tag-name-filter cat -- --all &&
	git show-ref
) &&
git clone file:///`pwd`/foo newFoo &&
(
	cd newFoo &&
	git show-ref
)
-- >8 --

which produces the result:

 ...
 07fa3d4c3aba83b293e24a12a87faca363ad34e6 refs/heads/master
 90a32c29df06b4f6e4218c1b82b35a3e49aed9f2 refs/original/refs/heads/master
 90a32c29df06b4f6e4218c1b82b35a3e49aed9f2 refs/original/refs/tags/tagfoo
 ef92777a31663135426a9648d0abb0c06b448fbe refs/tags/tagfoo
 ...
 Resolving deltas: 100% (1/1), done.
 error: refs/tags/tagfoo does not point to a valid object!
 07fa3d4c3aba83b293e24a12a87faca363ad34e6 refs/heads/master
 07fa3d4c3aba83b293e24a12a87faca363ad34e6 refs/remotes/origin/HEAD
 07fa3d4c3aba83b293e24a12a87faca363ad34e6 refs/remotes/origin/master
 $

The error message bisects to:

commit 5bdc32d3e50d8335c65e136e6b5234c5dd92a7a9 (tags/v1.6.5-rc3~20)
Author: Nicolas Pitre <nico@fluxnic.net>
Date:   Fri Sep 25 23:54:42 2009 -0400

    make 'git clone' ask the remote only for objects it cares about
    
    Current behavior of 'git clone' when not using --mirror is to fetch
    everything from the peer, and then filter out unwanted refs just before
    writing them out to the cloned repository.  This may become highly
    inefficient if the peer has an unusual ref namespace, or if it simply
    has "remotes" refs of its own, and those locally unwanted refs are
    connecting to a large set of objects which becomes unreferenced as soon
    as they are fetched.
    
    Let's filter out those unwanted refs from the peer _before_ asking it
    what refs we want to fetch instead, which is the most logical thing to
    do anyway.
    
    Signed-off-by: Nicolas Pitre <nico@fluxnic.net>
    Signed-off-by: Shawn O. Pearce <spearce@spearce.org>

That is not too surprising.  Before then, "git clone" fetched
all objects.

The error message is from refs.c::do_one_ref(), which notices the
missing object ef92777a31663135426a9648d0abb0c06b448fbe (i.e., ye
old tag object).

Call chain:

 cmd_clone() ->
  write_remote_refs() ->
   pack_refs() ->
    do_for_each_ref() ->
     do_one_ref()

The transport is done afaict by then.

Nico, Shawn: ideas?