git-svn clone behaves non-deterministic

git-svn clone behaves non-deterministic

[Josef Wolf]

Hello,

I am new to git and decided to get my feet wet by first cloning and
playing with my existing svn repositories.  Thus, I've done this:

 cd /my/test/repos
 for i in repo1 repo2 repo3; do
   repos=https://my.repos.server/repos/$i/trunk
   svn co        $repos svn/$i
   git-svn clone $repos git/$i
 done

 for i in `cd svn; echo *`; do diff --exclude /.svn -Nruw */$i; done

With this, I see that four of the repositories are cloned as expected,
but the fifth has only the .git directory in it.  It appears that
the clone command stopped cloning at r2008, while the repository is
currently at r3761.  So almost the half of the history was not
cloned at all.  I've investigated the offending revision and the
revisions around it, but I can't see anything special about them.
The effect is perfectly reproducible and it stops always on the very
same revision.  I get no error message at all.  I've attached the
last lines of the clone operation at the end of this mail.

Then I go to another machine and enter exactly the same commands as
above.  Both machines are fresh ubuntu-8.10 default installs.

To my surprise, on this other machine the clone operation seems to
have worked for all the repositories.  But the diff command shows
me that arbitrary files are missing in _all_ of the repositories.

Since the repositories were not modified in the mean time, it looks
to me like there's a problem with the "git-svn clone" command.

Any ideas?

  r2005 = e487699eb8f94d6c5e9b732ba75e1c50cae3bb5e (git-svn)
          A       misc/ubuntu-remaster/Rakefile
  r2006 = 8b15a1a5bdc8e6a3bfd7e6eb717ef4c2db0f1dbf (git-svn)
          M       misc/ubuntu-remaster/Rakefile
  r2007 = 27678a3bd98400abdaaaf9af543f9558593fc224 (git-svn)
          M       misc/ubuntu-remaster/Rakefile
  r2008 = bbc2d0cbe4419a499d29afda281b85b5e744a5e6 (git-svn)
  Auto packing your repository for optimum performance. You may also
  run "git gc" manually. See "git help gc" for more information.
  Counting objects: 12446, done.
  Compressing objects: 100% (11999/11999), done.
  Writing objects: 100% (12446/12446), done.
  Total 12446 (delta 7657), reused 0 (delta 0)
  jw@vdr1:/my/test/repos$ l git/misc/
  total 12
  drwxr-xr-x 3 jw jw 4096 Nov 27 08:27 ./
  drwxr-xr-x 3 jw jw 4096 Nov 27 08:27 ../
  drwxr-xr-x 9 jw jw 4096 Nov 27 08:38 .git/

Re: git-svn clone behaves non-deterministic

[Michael J Gruber]

Josef Wolf venit, vidit, dixit 27.11.2008 08:53:
> Hello,
> 
> I am new to git and decided to get my feet wet by first cloning and
> playing with my existing svn repositories.  Thus, I've done this:
> 
>  cd /my/test/repos
>  for i in repo1 repo2 repo3; do
>    repos=https://my.repos.server/repos/$i/trunk
>    svn co        $repos svn/$i
>    git-svn clone $repos git/$i
>  done
> 
>  for i in `cd svn; echo *`; do diff --exclude /.svn -Nruw */$i; done
> 
> With this, I see that four of the repositories are cloned as expected,
> but the fifth has only the .git directory in it.  It appears that
> the clone command stopped cloning at r2008, while the repository is
> currently at r3761.  So almost the half of the history was not
> cloned at all.  I've investigated the offending revision and the
> revisions around it, but I can't see anything special about them.
> The effect is perfectly reproducible and it stops always on the very
> same revision.  I get no error message at all.  I've attached the
> last lines of the clone operation at the end of this mail.

First of all: What does git --version say? Is it self-compiled git or
git as it comes with U 8.10?

Then: I assume the above is not quite what you did, otherwise I would be
surprised you see a 4th and 5th clone...

Your repo is probably one giant svn repo. In any case, the fact that the
repo is at $rev does not imply that the last commit on trunk is at $rev.
 What is the last commit on trunk?

Other than that: There are so many sins you can commit (!) in an svn
repo that it it is hard to tell which one threw git-svn off. Can you
share the repo?

> Then I go to another machine and enter exactly the same commands as
> above.  Both machines are fresh ubuntu-8.10 default installs.
> 
> To my surprise, on this other machine the clone operation seems to
> have worked for all the repositories.  But the diff command shows
> me that arbitrary files are missing in _all_ of the repositories.

I don't think computers behave deterministically, at least not according
to my experience. After all they rely on quantum mechanics, and who has
ever understood the measurement process, the collapsing of the wave
function?

OK: Whenever I got 2 different results after doing the same twice I
found that indeed some hidden/forgotten variable was not the same... Any
chance you find one?

Michael

Re: git-svn clone behaves non-deterministic

[Josef Wolf]

Thanks for your reply, Michael!

On Thu, Nov 27, 2008 at 10:55:12AM +0100, Michael J Gruber wrote:
> Josef Wolf venit, vidit, dixit 27.11.2008 08:53:
> > Hello,
> > 
> > I am new to git and decided to get my feet wet by first cloning and
> > playing with my existing svn repositories.  Thus, I've done this:
> > 
> >  cd /my/test/repos
> >  for i in repo1 repo2 repo3; do
> >    repos=https://my.repos.server/repos/$i/trunk
> >    svn co        $repos svn/$i
> >    git-svn clone $repos git/$i
> >  done
> > 
> >  for i in `cd svn; echo *`; do diff --exclude /.svn -Nruw */$i; done
> > 
> > With this, I see that four of the repositories are cloned as expected,
> > but the fifth has only the .git directory in it.  It appears that
> > the clone command stopped cloning at r2008, while the repository is
> > currently at r3761.  So almost the half of the history was not
> > cloned at all.  I've investigated the offending revision and the
> > revisions around it, but I can't see anything special about them.
> > The effect is perfectly reproducible and it stops always on the very
> > same revision.  I get no error message at all.  I've attached the
> > last lines of the clone operation at the end of this mail.
> 
> First of all: What does git --version say? Is it self-compiled git or
> git as it comes with U 8.10?

This is git-1.5.6.3 as it comes with U-8.10.

> Then: I assume the above is not quite what you did, otherwise I would be
> surprised you see a 4th and 5th clone...

That's correct. :-)  I have actually changed the names and number of the
repositories to avoid disclosing too much privacy ;-)

> Your repo is probably one giant svn repo.

This are 5 repositories in total.  The repos with the problem above is
the biggest of them (about 80MB, 3761 revisions).  But on the other
machine the other repositories also have missing files.

> In any case, the fact that the
> repo is at $rev does not imply that the last commit on trunk is at $rev.
> What is the last commit on trunk?

While I have created the recommended tags/branches/trunk layout to be
prepared, there have never been commits to tags or branches.  A long
time ago (at revisions 83..90) there were commits to /vendor, but after
r90 only trunk was ever used.

So all commits from r90 on were commits to trunk and none of the
revisions since r2008 up to HEAD ever touched anything outside trunk.

> Other than that: There are so many sins you can commit (!) in an svn
> repo that it it is hard to tell which one threw git-svn off. Can you
> share the repo?

Unfortunately not. :-(

> > Then I go to another machine and enter exactly the same commands as
> > above.  Both machines are fresh ubuntu-8.10 default installs.
> > 
> > To my surprise, on this other machine the clone operation seems to
> > have worked for all the repositories.  But the diff command shows
> > me that arbitrary files are missing in _all_ of the repositories.
> 
> I don't think computers behave deterministically, at least not according
> to my experience. After all they rely on quantum mechanics, and who has
> ever understood the measurement process, the collapsing of the wave
> function?

Although everything relies on quantum mechanics, common experience shows
that our world is much more deterministic than one would expect from
quantum theory.  Have you ever seen a cat being dead and alive at the
same time? ;-)

> OK: Whenever I got 2 different results after doing the same twice I
> found that indeed some hidden/forgotten variable was not the same...
> Any chance you find one?

Well, I am new to git, so I was searching for help/guidance on the list.

BTW: I just found the git-fsck command:

     Checking HEAD link
     notice: HEAD points to an unborn branch (master)
     Checking object directory
     Checking reflog 0000..0000->e23f..6f50
     [ 1998 more "Checking reflog" files deleted ]
     Checking reflog 273e..5c24->43bb..aee6
     Checking connectivity (32 objects)
     Checking 03ea6bbee94997ab9ddaecb3088b460eb4b87636
     Checking edcb1771f35ff085b7dfd546bcc51938f15a05d0
     Checking d7c3ec4b134ebf151cf2e78f6af729ed3d345e0d

The output of this command looks a little bit strange to me as a
newcomer:

- "HEAD points to unborn branch" confirms that the import failed
- there are exactly 2000 "Checking reflog" lines.  Since the last
  imported revision was 2008, I would have expected 2008 (or 2007
  or 2009) such lines.  My first thought was this might be related
  to the --repack option defaulting to 1000.  But with --repack=570
  it also stops after r2008.
- the "Checking connectivity" line mentions 32 objects, but only
  three "Checking" lines are listed after that.

Re: git-svn clone behaves non-deterministic

[Josef Wolf]

On Thu, Nov 27, 2008 at 07:29:47PM +0100, Josef Wolf wrote:

> - "HEAD points to unborn branch" confirms that the import failed
> - there are exactly 2000 "Checking reflog" lines.  Since the last
>   imported revision was 2008, I would have expected 2008 (or 2007
>   or 2009) such lines.  My first thought was this might be related
>   to the --repack option defaulting to 1000.  But with --repack=570
>   it also stops after r2008.

I've done lots of trial-and-error and I think I've finally found the
point where the script stops working.  Fortunately, git-svn is a perl
script :-). So I've found in the sub gs_do_update, the
statement

   $reporter->finish_report($pool);

is the last executed function.  Here is what Data::Dumper says about
the $pool parameter:

  $VAR1 = bless(
              do {
                  \(
                      my $o = bless(
                         do{
                              \(my $o = 148559280)
                         },
                         '_p_apr_pool_t'
                      )
                   )
              },
              'SVN::Pool'
          );

I have tried to format the Data::Dumper output more readable.

Any hints how to get closer to the problem?