[PATCH] git-svn: do a partial rebuild if rev_map is out-of-date

[PATCH] git-svn: do a partial rebuild if rev_map is out-of-date

[Deskin Miller]

Suppose you're using git-svn to work with a certain SVN repository.
Since you don't like 'git-svn fetch' to take forever, and you don't want
to accidentally interrupt it and end up corrupting your repository, you
set up a remote Git repository to mirror the SVN repository, which does
its own 'git-svn fetch' on a cronjob; now you can 'git-fetch' from the
Git mirror into your local repository, and still dcommit to SVN when you
have changes to push.

After you do this, though, git-svn will get very confused if you ever
try to do 'git-svn fetch' in your local repository again, since its
rev_map will differ from the branch's head, and it will be unable to
fetch new commits from SVN because of the metadata conflict.  But all
the necessary metadata are there in the Git commit message; git-svn
already knows how to rebuild rev_map files that get blown away, by
using the metadata.

This commit will have git-svn do a partial rebuild of the rev_map to
match the true state of the branch, if it ever is used to fetch again.

This will only work for projects not using either noMetadata or
useSvmProps configuration options; if you are using these options,
git-svn will fall back to the previous behaviour.

Signed-off-by: Deskin Miller <deskinm@umich.edu>
---
It's possible that this can be extended to work with useSvmProps, but I don't
know how to do so.

 git-svn.perl |   23 ++++++++++++++++++-----
 1 files changed, 18 insertions(+), 5 deletions(-)

diff --git a/git-svn.perl b/git-svn.perl
index 88066c9..2e7a8ce 100755
--- a/git-svn.perl
+++ b/git-svn.perl
@@ -2626,9 +2626,9 @@ sub rebuild_from_rev_db {
 sub rebuild {
 	my ($self) = @_;
 	my $map_path = $self->map_path;
-	return if (-e $map_path && ! -z $map_path);
+	my $partial = (-e $map_path && ! -z $map_path);
 	return unless ::verify_ref($self->refname.'^0');
-	if ($self->use_svm_props || $self->no_metadata) {
+	if (!$partial && ($self->use_svm_props || $self->no_metadata)) {
 		my $rev_db = $self->rev_db_path;
 		$self->rebuild_from_rev_db($rev_db);
 		if ($self->use_svm_props) {
@@ -2638,10 +2638,12 @@ sub rebuild {
 		$self->unlink_rev_db_symlink;
 		return;
 	}
-	print "Rebuilding $map_path ...\n";
+	print "Rebuilding $map_path ...\n" if (!$partial);
+	my ($base_rev, $head) = ($partial ? $self->rev_map_max_norebuild(1) :
+		(undef, undef));
 	my ($log, $ctx) =
 	    command_output_pipe(qw/rev-list --pretty=raw --no-color --reverse/,
-	                        $self->refname, '--');
+				($head ? "^$head" : ""), $self->refname, '--');
 	my $metadata_url = $self->metadata_url;
 	remove_username($metadata_url);
 	my $svn_uuid = $self->ra_uuid;
@@ -2664,12 +2666,17 @@ sub rebuild {
 		    ($metadata_url && $url && ($url ne $metadata_url))) {
 			next;
 		}
+		if ($partial && $head) {
+			print "Partial-rebuilding $map_path ...\n";
+			print "Currently at $base_rev = $head\n";
+			$head = undef;
+		}
 
 		$self->rev_map_set($rev, $c);
 		print "r$rev = $c\n";
 	}
 	command_close_pipe($log, $ctx);
-	print "Done rebuilding $map_path\n";
+	print "Done rebuilding $map_path\n" if (!$partial || !$head);
 	my $rev_db_path = $self->rev_db_path;
 	if (-f $self->rev_db_path) {
 		unlink $self->rev_db_path or croak "unlink: $!";
@@ -2809,6 +2816,12 @@ sub rev_map_set {
 sub rev_map_max {
 	my ($self, $want_commit) = @_;
 	$self->rebuild;
+	my ($r, $c) = $self->rev_map_max_norebuild($want_commit);
+	$want_commit ? ($r, $c) : $r;
+}
+
+sub rev_map_max_norebuild {
+	my ($self, $want_commit) = @_;
 	my $map_path = $self->map_path;
 	stat $map_path or return $want_commit ? (0, undef) : 0;
 	sysopen(my $fh, $map_path, O_RDONLY) or croak "open: $!";
-- 
1.5.4.3

Re: [PATCH] git-svn: do a partial rebuild if rev_map is out-of-date

[Eric Wong]

Deskin Miller <deskinm@umich.edu> wrote:
> This commit will have git-svn do a partial rebuild of the rev_map to
> match the true state of the branch, if it ever is used to fetch again.
> 
> This will only work for projects not using either noMetadata or
> useSvmProps configuration options; if you are using these options,
> git-svn will fall back to the previous behaviour.
> 
> Signed-off-by: Deskin Miller <deskinm@umich.edu>
> ---

Hi Deskin,

This seems to break the following test case for me:

*** t9107-git-svn-migrate.sh ***
*   ok 1: setup old-looking metadata
*   ok 2: git-svn-HEAD is a real HEAD
*   ok 3: initialize old-style (v0) git svn layout
*   ok 4: initialize a multi-repository repo
*   ok 5: multi-fetch works on partial urls + paths
*   ok 6: migrate --minimize on old inited layout
* FAIL 7: .rev_db auto-converted to .rev_map.UUID

I haven't had time to diagnose it.  Also, can you add a test that
demonstrates this functionality (and ensures things keeps working when
future work is done on git-svn?)

Thanks.

-- 
Eric Wong

[PATCH v2 0/1] git-svn: testcase for partial rebuild

[Deskin Miller]

>From 45c9876a04ff0aac141300dd10fca50d6db30522 Mon Sep 17 00:00:00 2001
From: Deskin Miler <deskinm@umich.edu>
Date: Thu, 18 Sep 2008 17:55:14 -0400

Signed-off-by: Deskin Miller <deskinm@umich.edu>
---
On Wed, Sep 17, 2008 at 11:38:04PM -0700, Eric Wong wrote:
> This seems to break the following test case for me:
> 
> *** t9107-git-svn-migrate.sh ***
> *   ok 1: setup old-looking metadata
> *   ok 2: git-svn-HEAD is a real HEAD
> *   ok 3: initialize old-style (v0) git svn layout
> *   ok 4: initialize a multi-repository repo
> *   ok 5: multi-fetch works on partial urls + paths
> *   ok 6: migrate --minimize on old inited layout
> * FAIL 7: .rev_db auto-converted to .rev_map.UUID
> 
> I haven't had time to diagnose it.  Also, can you add a test that
> demonstrates this functionality (and ensures things keeps working when
> future work is done on git-svn?)

Thanks for the response; I had a bug in my Perl that my testing hadn't caught.
Gave me an opportunity to learn how the git testsuites work!

This testcase fails for me when applied to master, and passes with patch 1/1 in
the series.

 t/t9126-git-svn-partial-rebuild.sh |   53 ++++++++++++++++++++++++++++++++++++
 1 files changed, 53 insertions(+), 0 deletions(-)
 create mode 100755 t/t9126-git-svn-partial-rebuild.sh

diff --git a/t/t9126-git-svn-partial-rebuild.sh b/t/t9126-git-svn-partial-rebuild.sh
new file mode 100755
index 0000000..9a94866
--- /dev/null
+++ b/t/t9126-git-svn-partial-rebuild.sh
@@ -0,0 +1,53 @@
+#!/bin/sh
+#
+# Copyright (c) 2008 Deskin Miller
+#
+
+test_description='git svn partial-rebuild tests'
+. ./lib-git-svn.sh
+
+test_expect_success \
+    'initialize svnrepo' '
+	mkdir import &&
+	cd import &&
+	mkdir trunk branches tags &&
+	cd trunk &&
+	echo foo > foo &&
+	cd .. &&
+	svn import -m "import for git-svn" . "$svnrepo" >/dev/null &&
+	svn copy "$svnrepo"/trunk "$svnrepo"/branches/a \
+		-m "created branch a" &&
+	cd .. &&
+	rm -rf import &&
+	svn co "$svnrepo"/trunk trunk &&
+	cd trunk &&
+	echo bar >> foo &&
+	svn ci -m "updated trunk" &&
+	cd .. &&
+	svn co "$svnrepo"/branches/a a &&
+	cd a &&
+	echo baz >> a &&
+	svn add a &&
+	svn ci -m "updated a" &&
+	cd .. &&
+	git svn init --stdlayout "$svnrepo"'
+
+test_expect_success 'import an early SVN revision into git' \
+	'git svn fetch -r1:2'
+
+test_expect_success 'make full git mirror of SVN' \
+	'mkdir mirror &&
+	cd mirror &&
+	git init &&
+	git svn init --stdlayout "$svnrepo" &&
+	git svn fetch &&
+	cd ..'
+
+test_expect_success 'fetch from git mirror and partial-rebuild' \
+	'git config --add remote.origin.url "file://$PWD/mirror/.git" &&
+	git config --add remote.origin.fetch refs/remotes/*:refs/remotes/* &&
+	git fetch origin &&
+	git svn fetch
+	'
+
+test_done
-- 
1.6.0.2.GIT

Re: [PATCH v2 0/1] git-svn: testcase for partial rebuild

[Eric Wong]

Deskin Miller <deskinm@umich.edu> wrote:
> ---
> On Wed, Sep 17, 2008 at 11:38:04PM -0700, Eric Wong wrote:
> > This seems to break the following test case for me:
> > 
> > *** t9107-git-svn-migrate.sh ***
> > *   ok 1: setup old-looking metadata
> > *   ok 2: git-svn-HEAD is a real HEAD
> > *   ok 3: initialize old-style (v0) git svn layout
> > *   ok 4: initialize a multi-repository repo
> > *   ok 5: multi-fetch works on partial urls + paths
> > *   ok 6: migrate --minimize on old inited layout
> > * FAIL 7: .rev_db auto-converted to .rev_map.UUID
> > 
> > I haven't had time to diagnose it.  Also, can you add a test that
> > demonstrates this functionality (and ensures things keeps working when
> > future work is done on git-svn?)
> 
> Thanks for the response; I had a bug in my Perl that my testing hadn't caught.
> Gave me an opportunity to learn how the git testsuites work!
> 
> This testcase fails for me when applied to master, and passes with patch 1/1 in
> the series.

Thanks Deskin, this series is
  Acked-by: Eric Wong <normalperson@yhbt.net>

Junio: I would apply the patch series in reverse order to not break
tests on potential bisections, however.

-- 
Eric Wong

[PATCH v2 1/1] git-svn: do a partial rebuild if rev_map is out-of-date

[Deskin Miller]

>From 11ef3a043bb9cf89cd87d7030c684a1ee566a87a Mon Sep 17 00:00:00 2001
From: Deskin Miller <deskinm@umich.edu>
Date: Mon, 15 Sep 2008 21:12:58 -0400

Suppose you're using git-svn to work with a certain SVN repository.
Since you don't like 'git-svn fetch' to take forever, and you don't want
to accidentally interrupt it and end up corrupting your repository, you
set up a remote Git repository to mirror the SVN repository, which does
its own 'git-svn fetch' on a cronjob; now you can 'git-fetch' from the
Git mirror into your local repository, and still dcommit to SVN when you
have changes to push.

After you do this, though, git-svn will get very confused if you ever
try to do 'git-svn fetch' in your local repository again, since its
rev_map will differ from the branch's head, and it will be unable to
fetch new commits from SVN because of the metadata conflict.  But all
the necessary metadata are there in the Git commit message; git-svn
already knows how to rebuild rev_map files that get blown away, by
using the metadata.

This commit will have git-svn do a partial rebuild of the rev_map to
match the true state of the branch, if it ever is used to fetch again.

This will only work for projects not using either noMetadata or
useSvmProps configuration options; if you are using these options,
git-svn will fall back to the previous behaviour.

Signed-off-by: Deskin Miller <deskinm@umich.edu>
---
On Wed, Sep 17, 2008 at 11:38:04PM -0700, Eric Wong wrote:
> This seems to break the following test case for me:
> 
> *** t9107-git-svn-migrate.sh ***
> *   ok 1: setup old-looking metadata
> *   ok 2: git-svn-HEAD is a real HEAD
> *   ok 3: initialize old-style (v0) git svn layout
> *   ok 4: initialize a multi-repository repo
> *   ok 5: multi-fetch works on partial urls + paths
> *   ok 6: migrate --minimize on old inited layout
> * FAIL 7: .rev_db auto-converted to .rev_map.UUID
> 
> I haven't had time to diagnose it.  Also, can you add a test that
> demonstrates this functionality (and ensures things keeps working when
> future work is done on git-svn?)

Here is the new, fixed version.  It allows the test in patch 0/1 to pass.

 git-svn.perl |   24 +++++++++++++++++++-----
 1 files changed, 19 insertions(+), 5 deletions(-)

diff --git a/git-svn.perl b/git-svn.perl
index af8279a..80a5728 100755
--- a/git-svn.perl
+++ b/git-svn.perl
@@ -2626,9 +2626,9 @@ sub rebuild_from_rev_db {
 sub rebuild {
 	my ($self) = @_;
 	my $map_path = $self->map_path;
-	return if (-e $map_path && ! -z $map_path);
+	my $partial = (-e $map_path && ! -z $map_path);
 	return unless ::verify_ref($self->refname.'^0');
-	if ($self->use_svm_props || $self->no_metadata) {
+	if (!$partial && ($self->use_svm_props || $self->no_metadata)) {
 		my $rev_db = $self->rev_db_path;
 		$self->rebuild_from_rev_db($rev_db);
 		if ($self->use_svm_props) {
@@ -2638,10 +2638,13 @@ sub rebuild {
 		$self->unlink_rev_db_symlink;
 		return;
 	}
-	print "Rebuilding $map_path ...\n";
+	print "Rebuilding $map_path ...\n" if (!$partial);
+	my ($base_rev, $head) = ($partial ? $self->rev_map_max_norebuild(1) :
+		(undef, undef));
 	my ($log, $ctx) =
 	    command_output_pipe(qw/rev-list --pretty=raw --no-color --reverse/,
-	                        $self->refname, '--');
+				($head ? "$head.." : "") . $self->refname,
+				'--');
 	my $metadata_url = $self->metadata_url;
 	remove_username($metadata_url);
 	my $svn_uuid = $self->ra_uuid;
@@ -2664,12 +2667,17 @@ sub rebuild {
 		    ($metadata_url && $url && ($url ne $metadata_url))) {
 			next;
 		}
+		if ($partial && $head) {
+			print "Partial-rebuilding $map_path ...\n";
+			print "Currently at $base_rev = $head\n";
+			$head = undef;
+		}
 
 		$self->rev_map_set($rev, $c);
 		print "r$rev = $c\n";
 	}
 	command_close_pipe($log, $ctx);
-	print "Done rebuilding $map_path\n";
+	print "Done rebuilding $map_path\n" if (!$partial || !$head);
 	my $rev_db_path = $self->rev_db_path;
 	if (-f $self->rev_db_path) {
 		unlink $self->rev_db_path or croak "unlink: $!";
@@ -2809,6 +2817,12 @@ sub rev_map_set {
 sub rev_map_max {
 	my ($self, $want_commit) = @_;
 	$self->rebuild;
+	my ($r, $c) = $self->rev_map_max_norebuild($want_commit);
+	$want_commit ? ($r, $c) : $r;
+}
+
+sub rev_map_max_norebuild {
+	my ($self, $want_commit) = @_;
 	my $map_path = $self->map_path;
 	stat $map_path or return $want_commit ? (0, undef) : 0;
 	sysopen(my $fh, $map_path, O_RDONLY) or croak "open: $!";
-- 
1.6.0.2.GIT