[PATCH v3] git-svn: memoize _rev_list and rebuild

[PATCH v3] git-svn: memoize _rev_list and rebuild

[manjian2006]

From: lin zuojian <manjian2006@gmail.com>

According to profile data, _rev_list and rebuild consume a large
portion of time.  Memoize the results of _rev_list and memoize
rebuild internals to avoid subprocess invocation.

When importing 15152 revisions on a LAN, time improved from 10
hours to 3-4 hours.

Signed-off-by: lin zuojian <manjian2006@gmail.com>
---
 perl/Git/SVN.pm | 41 ++++++++++++++++++++++++++++++++++++++---
 1 file changed, 38 insertions(+), 3 deletions(-)

diff --git a/perl/Git/SVN.pm b/perl/Git/SVN.pm
index 5273ee8..6e804a2 100644
--- a/perl/Git/SVN.pm
+++ b/perl/Git/SVN.pm
@@ -1599,6 +1599,7 @@ sub tie_for_persistent_memoization {
 		my %lookup_svn_merge_cache;
 		my %check_cherry_pick_cache;
 		my %has_no_changes_cache;
+		my %_rev_list_cache;
 
 		tie_for_persistent_memoization(\%lookup_svn_merge_cache,
 		    "$cache_path/lookup_svn_merge");
@@ -1620,6 +1621,14 @@ sub tie_for_persistent_memoization {
 			SCALAR_CACHE => ['HASH' => \%has_no_changes_cache],
 			LIST_CACHE => 'FAULT',
 		;
+
+		tie_for_persistent_memoization(\%_rev_list_cache,
+		    "$cache_path/_rev_list");
+		memoize '_rev_list',
+			SCALAR_CACHE => 'FAULT',
+			LIST_CACHE => ['HASH' => \%_rev_list_cache],
+		;
+
 	}
 
 	sub unmemoize_svn_mergeinfo_functions {
@@ -1629,6 +1638,7 @@ sub tie_for_persistent_memoization {
 		Memoize::unmemoize 'lookup_svn_merge';
 		Memoize::unmemoize 'check_cherry_pick';
 		Memoize::unmemoize 'has_no_changes';
+		Memoize::unmemoize '_rev_list';
 	}
 
 	sub clear_memoized_mergeinfo_caches {
@@ -1959,11 +1969,25 @@ sub rebuild_from_rev_db {
 	unlink $path or croak "unlink: $!";
 }
 
+#define a global associate map to record rebuild status
+my %rebuild_status;
+#define a global associate map to record rebuild verify status
+my %rebuild_verify_status;
+
 sub rebuild {
 	my ($self) = @_;
 	my $map_path = $self->map_path;
 	my $partial = (-e $map_path && ! -z $map_path);
-	return unless ::verify_ref($self->refname.'^0');
+	my $verify_key = $self->refname.'^0';
+	if (!$rebuild_verify_status{$verify_key}) {
+		my $verify_result = ::verify_ref($verify_key);
+		if ($verify_result) {
+			$rebuild_verify_status{$verify_key} = 1;
+		}
+	}
+	if (!$rebuild_verify_status{$verify_key}) {
+		return;
+	}
 	if (!$partial && ($self->use_svm_props || $self->no_metadata)) {
 		my $rev_db = $self->rev_db_path;
 		$self->rebuild_from_rev_db($rev_db);
@@ -1977,10 +2001,21 @@ sub rebuild {
 	print "Rebuilding $map_path ...\n" if (!$partial);
 	my ($base_rev, $head) = ($partial ? $self->rev_map_max_norebuild(1) :
 		(undef, undef));
+	my $key_value = ($head ? "$head.." : "") . $self->refname;
+	if (exists $rebuild_status{$key_value}) {
+		print "Done rebuilding $map_path\n" if (!$partial || !$head);
+		my $rev_db_path = $self->rev_db_path;
+		if (-f $self->rev_db_path) {
+			unlink $self->rev_db_path or croak "unlink: $!";
+		}
+		$self->unlink_rev_db_symlink;
+		return;
+	}
 	my ($log, $ctx) =
-	    command_output_pipe(qw/rev-list --pretty=raw --reverse/,
-				($head ? "$head.." : "") . $self->refname,
+		command_output_pipe(qw/rev-list --pretty=raw --reverse/,
+				$key_value,
 				'--');
+	$rebuild_status{$key_value} = 1;
 	my $metadata_url = $self->metadata_url;
 	remove_username($metadata_url);
 	my $svn_uuid = $self->rewrite_uuid || $self->ra_uuid;
-- 
1.8.3.2

Re: [PATCH v3] git-svn: memoize _rev_list and rebuild

[Eric Wong]

manjian2006@gmail.com wrote:
> According to profile data, _rev_list and rebuild consume a large
> portion of time.  Memoize the results of _rev_list and memoize
> rebuild internals to avoid subprocess invocation.
> 
> When importing 15152 revisions on a LAN, time improved from 10
> hours to 3-4 hours.
> 
> Signed-off-by: lin zuojian <manjian2006@gmail.com>

Thanks!
Signed-off-by: Eric Wong <normalperson@yhbt.net>
Pushed for Junio.

The following changes since commit d9bb4be53bc5185244b4be9860562a012803bacb:

  Merge tag 'gitgui-0.19.0' of http://repo.or.cz/r/git-gui (2014-01-21 13:16:17 -0800)

are available in the git repository at:


  git://git.bogomips.org/git-svn.git master

for you to fetch changes up to ab0bcec9873f1fcef6c4b8825cc9e762c636ca9e:

  git-svn: memoize _rev_list and rebuild (2014-01-23 02:54:26 +0000)

----------------------------------------------------------------
lin zuojian (1):
      git-svn: memoize _rev_list and rebuild

 perl/Git/SVN.pm | 41 ++++++++++++++++++++++++++++++++++++++---
 1 file changed, 38 insertions(+), 3 deletions(-)

Re: [PATCH v3] git-svn: memoize _rev_list and rebuild

[Junio C Hamano]

Eric Wong <normalperson@yhbt.net> writes:

> Pushed for Junio.

Thanks.

Re: [PATCH v3] git-svn: memoize _rev_list and rebuild

[Eric Wong]

manjian2006@gmail.com wrote:
> From: lin zuojian <manjian2006@gmail.com>
> 
> According to profile data, _rev_list and rebuild consume a large
> portion of time.  Memoize the results of _rev_list and memoize
> rebuild internals to avoid subprocess invocation.

Hi, I am considering reverting _rev_list memoization because it
can increases memory usage a lot, which hurting fork() performance on
Linux.

Can you tell me if your original results are under Linux or another
kernel?  I may only disable _rev_list memoization under Linux and
leave other OSes unaffected, too.

I've merged some patches from Jakob to improve svn:mergeinfo
performance along with some followup patches to reduce memory
use.  However memory bloat is still a problem.

Currently in my "master" branch of git://bogomips.org/git-svn

Eric Wong (4):
      git-svn: reduce check_cherry_pick cache overhead
      git-svn: cache only mergeinfo revisions
      git-svn: clear global SVN pool between get_log invocations
      git-svn: remove mergeinfo rev caching

Jakob Stoklund Olesen (2):
      git-svn: only look at the new parts of svn:mergeinfo
      git-svn: only look at the root path for svn:mergeinfo