working with a large repository and git svn

working with a large repository and git svn

[Joe Corneli]

Greetings -

I am experiencing trouble with git svn, trying to import a
large repository (7.9 gigs, ~54000 commits) from Git into
SVN.

This has failed in a couple of different ways, depending
on the operating environment.  With Git version 1.7.3.5
running on Ubuntu 9.10, in the final step

  git svn dcommit --no-rebase

of the formula described below, I get:

 failing with "Can't fork at /usr/share/perl/5.10.0/Git.pm line 1261."

after committing just over 2000 revisions.

Previously, on Mac OS X 10.6.4 with git version 1.7.3.4,
it made it through about 18000 commits before failing with
some other error.  (I don't have that one recorded at the
moment.)

Separately from the latest attempt, I tried repacking the
repository before doing the "git svn" stuff, with

  git repack -a -d --depth=250 --window=250 -f

but that also failed ("pack-objects died of signal 11").

Any tips for dealing with new, large, repositories would
be appreciated.  The sequence of commands I used are below
the 8<.

Thanks,
Joe

8<-8<-8<-8<-8<-8<-8<-8<-8<-8<-8<-8<-8<-8<-8<-8<-8<-8<-8<-8<-8<

## Creating an svn repo

$ mkdir repo;
$ svnadmin create repo;
$ mkdir init;
$ touch init/README;
$ svn import init file://`pwd`/repo/init -m "Initial import";
$ svn checkout file://`pwd`/repo/init working;

## THIS PART FOLLOWS THE MODEL SUGGESTED BY THE FOLKS AT code.google.com
## (/Users/jac2349/planetary/destination/ IS THE LOCATION OF MY GIT REPO.)

$ mkdir cloning
$ cd cloning
$ git svn clone file:///Users/jac2349/planetary/repo/init
$ cd init
$ git fetch git:///Users/jac2349/planetary/destination/.git

$ git branch tmp $(cut -b-40 .git/FETCH_HEAD)
$ git tag -a -m "Last fetch" last tmp

$ INIT_COMMIT=$(git log tmp --pretty=format:%H | tail -1)
$ git checkout $INIT_COMMIT .
$ git commit -C $INIT_COMMIT

$ git rebase master tmp
$ git branch -M tmp master

$ git svn dcommit --no-rebase

$ mv .git/refs/tags/newlast .git/refs/tags/last

## BTW, THE --no-rebase FLAG KEEPS IT FROM BEING IMPOSSIBLY SLOW!

Re: working with a large repository and git svn

[Wesley J. Landaker]

On Tuesday, January 11, 2011 18:27:10 Joe Corneli wrote:
> I am experiencing trouble with git svn, trying to import a
> large repository (7.9 gigs, ~54000 commits) from Git into
> SVN.
> 
> This has failed in a couple of different ways, depending
> on the operating environment.  With Git version 1.7.3.5
> running on Ubuntu 9.10, in the final step
> 
>   git svn dcommit --no-rebase
> 
> of the formula described below, I get:
> 
>  failing with "Can't fork at /usr/share/perl/5.10.0/Git.pm line 1261."
> 
> after committing just over 2000 revisions.

I haven't tried importing 8 GB from Git to Subversion, but I have used Git 
against existing huge Subversion repositories that are >= 10 GB with little 
trouble, other than that it takes forever because Subversion is slow.

Here are some thoughts on how I'd approach what you are doing. Realize that 
no matter what, it's still probably going to take "forever" (e.g. run it 
over the weekend).

  1) Sounds like git-svn is running out of resources on your machine -- 
that's probably a bug, but work around it: Don't dcommit all 20000 revisions 
at once. Maybe write a shell script that goes through and dcommits a 100 
commits at a time.

  2) Do you need the full history to be in SVN? Can you rebase/squash large 
parts together and thus need to commit less revisions in the first place?

  3) I love git-svn for working with Subversion repositories, but you could 
consider a different tool, like tailor, if you can't make git-svn do what 
you want. I have also heard talk (but I don't know the state of things) of 
people working on a fast-import tool for SVN, so you could git-fast-export 
and svn-fast-import in a big batch.

  4) Does 8 GB of data really belong in the same repository? Maybe it should 
really be split up and used with git submodules or SVN externals? That may 
make things easier to work with in the long term.

  5) Do you really want to be going from Git, to Subversion? That seems like 
a big step backwards. =)

In any case, good luck!

Re: working with a large repository and git svn

[Joe Corneli]

>  1) Sounds like git-svn is running out of resources on your machine --
> that's probably a bug, but work around it: Don't dcommit all 20000 revisions
> at once. Maybe write a shell script that goes through and dcommits a 100
> commits at a time.

Hm, I found a related blog post here, but designed for interactive use:
http://fredericiana.com/2009/12/31/partial-svn-dcommit-with-git/
Could you give me a more detailed hint about how to do what you suggested?

>  2) Do you need the full history to be in SVN? Can you rebase/squash large
> parts together and thus need to commit less revisions in the first place?

Maybe.  We want a tool for managing the entire history, and Git seems
like a good tool for that.  At the same time, checking out the entire
history can take a long time - if we could just check out just the
latest files and check them back in in a sensible way, that would be
good - SVN does seem suitable for that purpose.  If there's a git-only
way to do this I'd be happy to know about that as well!

>  3) I love git-svn for working with Subversion repositories, but you could
> consider a different tool, like tailor, if you can't make git-svn do what
> you want.

Tried it, but it didn't even get through the initiation phase.  I
asked for help in the relevant mailing list.

> people working on a fast-import tool for SVN, so you could git-fast-export
> and svn-fast-import in a big batch.

Not finding these.

>  4) Does 8 GB of data really belong in the same repository? Maybe it should
> really be split up and used with git submodules or SVN externals? That may
> make things easier to work with in the long term.

Probably true.  if there was a nice way to give each *file* its own
associated "repository", then stitch these together into packets (even
"on demand"), that would be cool.  I was assuming we could do fancy
stuff like this as "future work" however - and it would seem that if
we use a completely git-based solution we'll be there.

>  5) Do you really want to be going from Git, to Subversion? That seems like
> a big step backwards. =)

If there's a good way to just pull down the latest revision into a
working copy and be able to push that back to the repo that would be
nice.  This doesn't seem to be the Git way, but for an 8 gig repo it's
probably pretty important feature.  Thoughts?

Thanks,
Joe

Re: working with a large repository and git svn

[Jonathan Nieder]

Wesley J. Landaker wrote:

>   3) I love git-svn for working with Subversion repositories, but you could 
> consider a different tool, like tailor, if you can't make git-svn do what 
> you want. I have also heard talk (but I don't know the state of things) of 
> people working on a fast-import tool for SVN, so you could git-fast-export 
> and svn-fast-import in a big batch.

I think the state of the art is currently git2svn[1] + "svnrdump load".
This requires permission to change properties on the svn repo, just
like svnsync would.

Hope that helps,
Jonathan

[1] http://repo.or.cz/w/git2svn.git

Re: working with a large repository and git svn

[Joe Corneli]

> I think the state of the art is currently git2svn

Thanks, that did indeed work, though, for the record it uses committer
name and email in the log that it generates, not author name and
email, but no worries!

Joe

Re: working with a large repository and git svn

[Ramkumar Ramachandra]

Hi Joe,

Joe Corneli writes:
> > I think the state of the art is currently git2svn
> 
> Thanks, that did indeed work, though, for the record it uses committer
> name and email in the log that it generates, not author name and
> email, but no worries!

That should be easy enough to fix with something like this (warning:
untested). A more elegant solution would actually use some sort of
user-configurable mapping from Git authors/ committers to SVN authors
though.

Signed-off-by: Ramkumar Ramachandra <artagnon@gmail.com>
--8<--
diff --git a/git2svn b/git2svn
index 2380775..3856696 100755
--- a/git2svn
+++ b/git2svn
@@ -261,12 +261,8 @@ COMMAND: while (!eof(IN)) {
 	    $commit{Mark} = $1;
 	    $next = next_line($IN);
 	}
-	if ($next =~ m/author +(.*)/) {
-	    $commit{Author} = $1;
-	    $next = next_line($IN);
-	}
-	unless ($next =~ m/committer +(.+) +<([^>]+)> +(\d+) +[+-](\d+)$/) {
-	    die "missing comitter: $_";
+	unless ($next =~ m/author +(.+) +<([^>]+)> +(\d+) +[+-](\d+)$/) {
+	    die "missing author: $_";
 	}
 
 	$commit{CommitterName} = $1;
@@ -275,6 +271,9 @@ COMMAND: while (!eof(IN)) {
 	$commit{CommitterTZ} = $4;
 
 	$next = next_line($IN);
+	if ($next =~ m/committer +(.*)/) {
+	    $next = next_line($IN);
+	}
 	my $log = read_data($IN, $next);
 
 	$next = next_line($IN);

Re: working with a large repository and git svn

[Jonathan Nieder]

Ramkumar Ramachandra wrote:
> Joe Corneli writes:

>>> I think the state of the art is currently git2svn
>>
>> Thanks, that did indeed work, though, for the record it uses committer
>> name and email in the log that it generates, not author name and
>> email, but no worries!
>
> That should be easy enough to fix with something like this (warning:
> untested). A more elegant solution would actually use some sort of
> user-configurable mapping from Git authors/ committers to SVN authors
> though.

Thanks for the cc.  (cc-ing lha, as I should have before.)

I suppose if svn will show only one of the two (committer and author)
then it is better to show the author.  Possible complications:

. The author lines in fast-import streams are optional.

. Existing users of the incremental import facility might not want the
  meaning of svn:author to change between imports.  _If_ that is a
  problem then a command-line option to switch behaviors might help.

. Is svn okay with non-monotonic dates?  (If not, then the committer
  date would need to be used.)

Modulo those complications I like the idea.  (Though I haven't read
the implementation, which follows for reference.)

> 
> Signed-off-by: Ramkumar Ramachandra <artagnon@gmail.com>
> --8<--
> diff --git a/git2svn b/git2svn
> index 2380775..3856696 100755
> --- a/git2svn
> +++ b/git2svn
> @@ -261,12 +261,8 @@ COMMAND: while (!eof(IN)) {
>  	    $commit{Mark} = $1;
>  	    $next = next_line($IN);
>  	}
> -	if ($next =~ m/author +(.*)/) {
> -	    $commit{Author} = $1;
> -	    $next = next_line($IN);
> -	}
> -	unless ($next =~ m/committer +(.+) +<([^>]+)> +(\d+) +[+-](\d+)$/) {
> -	    die "missing comitter: $_";
> +	unless ($next =~ m/author +(.+) +<([^>]+)> +(\d+) +[+-](\d+)$/) {
> +	    die "missing author: $_";
>  	}
>  
>  	$commit{CommitterName} = $1;
> @@ -275,6 +271,9 @@ COMMAND: while (!eof(IN)) {
>  	$commit{CommitterTZ} = $4;
>  
>  	$next = next_line($IN);
> +	if ($next =~ m/committer +(.*)/) {
> +	    $next = next_line($IN);
> +	}
>  	my $log = read_data($IN, $next);
>  
>  	$next = next_line($IN);

Re: working with a large repository and git svn

[Michael Haggerty]

On 01/14/2011 09:29 AM, Jonathan Nieder wrote:
> . Is svn okay with non-monotonic dates?  (If not, then the committer
>   date would need to be used.)

Subversion can tolerate non-monotonic dates with one caveat: it breaks
the find-revision-by-date feature (e.g., "svn update -r '{2010-12-25}'")
for the time intervals with non-monotonic dates.  This is a seldom-used
feature and therefore its sacrifice is often accepted, for example when
the history of the Subversion project itself was migrated into the
Apache project's Subversion repository.

Michael

-- 
Michael Haggerty
mhagger@alum.mit.edu
http://softwareswirl.blogspot.com/

[PATCH] Optionally parse author information

[Ramkumar Ramachandra]

When creating a new commit, instead of picking up the SVN author from
the committer's email, pick it up from the author's email, when
possible. Also add a new command-line switch '--ignore-author' to
force older behavior for backward compatibilty.

Noticed-by: Joe Corneli <holtzermann17@gmail.com>
Signed-off-by: Ramkumar Ramachandra <artagnon@gmail.com>
---
 git2svn |   25 +++++++++++++++++++------
 1 files changed, 19 insertions(+), 6 deletions(-)

diff --git a/git2svn b/git2svn
index 2380775..8ef55f1 100755
--- a/git2svn
+++ b/git2svn
@@ -36,7 +36,7 @@ use Pod::Usage;
 my $IN;
 my $OUT;
 
-my ($help, $verbose, $keeplogs, $no_load);
+my ($help, $verbose, $keeplogs, $no_load, $ignore_author);
 
 # svn
 my $svntree = "repro";
@@ -200,6 +200,7 @@ $result = GetOptions ("git-branch=s" => \$branch,
 		      "svn-prefix=s" => \$basedir,
 		      "keep-logs" => \$keeplogs,
 		      "no-load" => \$no_load,
+		      "ignore-author" => \$ignore_author,
 		      "verbose+" => \$verbose,
 		      "help" => \$help) or pod2usage(2);
 
@@ -261,12 +262,15 @@ COMMAND: while (!eof(IN)) {
 	    $commit{Mark} = $1;
 	    $next = next_line($IN);
 	}
-	if ($next =~ m/author +(.*)/) {
-	    $commit{Author} = $1;
+	if ($next =~ m/author +(.+) +<([^>]+)> +(\d+) +[+-](\d+)$/) {
+	    $commit{AuthorName} = $1;
+	    $commit{AuthorEmail} = $2;
+	    $commit{AuthorWhen} = $3;
+	    $commit{AuthorTZ} = $4;
 	    $next = next_line($IN);
 	}
 	unless ($next =~ m/committer +(.+) +<([^>]+)> +(\d+) +[+-](\d+)$/) {
-	    die "missing comitter: $_";
+	    die "missing committer: $_";
 	}
 
 	$commit{CommitterName} = $1;
@@ -291,11 +295,15 @@ COMMAND: while (!eof(IN)) {
 	    strftime("%Y-%m-%dT%H:%M:%S.000000Z", 
 		     gmtime($commit{CommitterWhen}));
 
-	my $author = "(no author)";
+	my $author = "git2svn-dump";
 	if ($commit{CommitterEmail} =~ m/([^@]+)/) {
 	    $author = $1;
 	}
-	$author = "git2svn-dump" if ($author eq "(no author)");
+	unless ($ignore_author) {
+	    if ($commit{AuthorEmail} =~ m/([^@]+)/) {
+	        $author = $1;
+	    }
+	}
 
 	my $props = "";
 	$props .= prop("svn:author", $author);
@@ -486,6 +494,11 @@ match the default GIT branch (master).
 
 Don't load the svn repository or update the syncpoint tagname.
 
+=item B<--ignore-author>
+
+Ignore "author" lines in the fast-import stream. Use "committer"
+information instead.
+
 =item B<--keep-logs>
 
 Don't delete the logs in $CWD/.data on success.
-- 
1.7.4.rc1.7.g2cf08.dirty

Re: [PATCH] Optionally parse author information

[Joe Corneli]

I tested it, and it seems to use email handle instead of author name
(perhaps that's intentional, though in my case it's not so desirable)
but, quite critically, it gets the dates wrong:

~/pmhistory.svn$ svn log -l 5
------------------------------------------------------------------------
r53127 | majordomo | 2011-01-10 18:31:58 -0500 (Mon, 10 Jan 2011) | 1 line


------------------------------------------------------------------------
r53126 | majordomo | 2011-01-10 18:31:58 -0500 (Mon, 10 Jan 2011) | 1 line


------------------------------------------------------------------------
r53125 | majordomo | 2011-01-10 18:31:58 -0500 (Mon, 10 Jan 2011) | 1 line


------------------------------------------------------------------------
r53124 | majordomo | 2011-01-10 18:31:57 -0500 (Mon, 10 Jan 2011) | 1 line


------------------------------------------------------------------------
r53123 | majordomo | 2011-01-10 18:31:57 -0500 (Mon, 10 Jan 2011) | 1 line

~/pmhistory.git$ git log -5
commit 411b8698e494ee12799300611fed0c8029e76ad3
Author: milogardner <majordomo@planetmath.org>
Date:   Thu Dec 16 14:11:57 2010 +0000

commit d12f8472cc06feec1a0e3a652e4ac14d7869fb3f
Author: milogardner <majordomo@planetmath.org>
Date:   Thu Dec 16 14:00:13 2010 +0000

commit 5febb4767563255280d95091ff9b2b0207042071
Author: Mathprof <majordomo@planetmath.org>
Date:   Wed Dec 15 23:02:47 2010 +0000

commit 61b11b97c4e503c353af5c1cd68e17b053d12b8e
Author: pahio <majordomo@planetmath.org>
Date:   Mon Dec 13 17:59:12 2010 +0000

commit 14d66e2dcd6151eb7214a9afbab159459912da6d
Author: pahio <majordomo@planetmath.org>
Date:   Mon Dec 13 17:54:21 2010 +0000

Re: [PATCH] Optionally parse author information

[Jonathan Nieder]

Joe Corneli wrote:

> I tested it, and it seems to use email handle instead of author name
> (perhaps that's intentional, though in my case it's not so desirable)

Good point.  Presumably git2svn is using the local part of the email
address to mimic svn's default behavior of using one's username.

Other possibilities:

 - email address (e.g., majordomo@planetmath.org, as in most google
   repositories)
 - display name (e.g., "Joe Corneli").  I don't know if svn-related
   tools or scripts assume that svn:author doesn't contain spaces.
 - full ident string (e.g., "Joe Corneli <majordomo@planetmath.org>")
 - whatever the operator wants (mapping specified in authors file).

My guess: an "authors file" facility would be needed to cover all
cases, but whichever rule you want to implement short of that could
also be useful.

> but, quite critically, it gets the dates wrong:
> 
> ~/pmhistory.svn$ svn log -l 5
> ------------------------------------------------------------------------
> r53127 | majordomo | 2011-01-10 18:31:58 -0500 (Mon, 10 Jan 2011) | 1 line
[...]
> ~/pmhistory.git$ git log -5
> commit 411b8698e494ee12799300611fed0c8029e76ad3
> Author: milogardner <majordomo@planetmath.org>
> Date:   Thu Dec 16 14:11:57 2010 +0000

Maybe it is using the committer date (as shown by "git log --format=fuller")
and someone rebased recently.  If you don't care about svn's '{date}'
construct working (meaning out-of-order dates are ok) then author date
might be more suitable.  Presumably the important thing is for it to
be consistent.

Hope that helps,
Jonathan

Re: [PATCH] Optionally parse author information

[Ramkumar Ramachandra]

Hi Joe,

Joe Corneli writes:
> I tested it, and it seems to use email handle instead of author name
> (perhaps that's intentional, though in my case it's not so desirable)
> but, quite critically, it gets the dates wrong:

Yes. I didn't change it's core behavior- it used to extract the
information from the committer's email address previously; I just
changed it to use the author's email address. For dates, it uses
committer dates again, and this is probably desirable: author dates
aren't necessarily monotonic, and this can break some functionality in
SVN.

Ofcourse, a lot more is possible with an patch that allows users to
configure all these things. Until then, I recommend that you just edit
the source to achieve the desired results.

-- Ram