[PATCH] git-svn: cache max revision in rev_db databases

[PATCH] git-svn: cache max revision in rev_db databases

[Sam Vilain]

Cache the maximum revision for each rev_db URL rather than looking it
up each time.  This saves a lot of time when rebuilding indexes on a
freshly cloned repository.

Signed-off-by: Sam Vilain <sam@vilain.net>
---
 git-svn.perl |    4 ++++
 1 files changed, 4 insertions(+), 0 deletions(-)

diff --git a/git-svn.perl b/git-svn.perl
index 610563c..c9758a0 100755
--- a/git-svn.perl
+++ b/git-svn.perl
@@ -804,6 +804,7 @@ sub working_head_info {
 	my ($head, $refs) = @_;
 	my ($fh, $ctx) = command_output_pipe('log', $head);
 	my $hash;
+	my %max;
 	while (<$fh>) {
 		if ( m{^commit ($::sha1)$} ) {
 			$hash = $1;
@@ -812,11 +813,14 @@ sub working_head_info {
 		next unless s{^\s+(git-svn-id:)}{$1};
 		my ($url, $rev, $uuid) = extract_metadata($_);
 		if (defined $url && defined $rev) {
+			next if $max{$url} and $max{$url} < $rev;
 			if (my $gs = Git::SVN->find_by_url($url)) {
 				my $c = $gs->rev_db_get($rev);
 				if ($c && $c eq $hash) {
 					close $fh; # break the pipe
 					return ($url, $rev, $uuid, $gs);
+				} else {
+					$max{$url} ||= $gs->rev_db_max;
 				}
 			}
 		}
-- 
1.5.0.4.210.gf8a7c-dirty

Re: [PATCH] git-svn: cache max revision in rev_db databases

[Eric Wong]

Sam Vilain <sam@vilain.net> wrote:
> Cache the maximum revision for each rev_db URL rather than looking it
> up each time.  This saves a lot of time when rebuilding indexes on a
> freshly cloned repository.

It looks correct, but I'm not sure how often we can even hit this
optimization to make it worth it.

Any live repositories and benchmarks you've run this on?
(without the rev-list => log patch applied, which would be a big
performance improvement if all log messages were under 16k).

> Signed-off-by: Sam Vilain <sam@vilain.net>
> ---
>  git-svn.perl |    4 ++++
>  1 files changed, 4 insertions(+), 0 deletions(-)
> 
> diff --git a/git-svn.perl b/git-svn.perl
> index 610563c..c9758a0 100755
> --- a/git-svn.perl
> +++ b/git-svn.perl
> @@ -804,6 +804,7 @@ sub working_head_info {
>  	my ($head, $refs) = @_;
>  	my ($fh, $ctx) = command_output_pipe('log', $head);
>  	my $hash;
> +	my %max;
>  	while (<$fh>) {
>  		if ( m{^commit ($::sha1)$} ) {
>  			$hash = $1;
> @@ -812,11 +813,14 @@ sub working_head_info {
>  		next unless s{^\s+(git-svn-id:)}{$1};
>  		my ($url, $rev, $uuid) = extract_metadata($_);
>  		if (defined $url && defined $rev) {
> +			next if $max{$url} and $max{$url} < $rev;
>  			if (my $gs = Git::SVN->find_by_url($url)) {
>  				my $c = $gs->rev_db_get($rev);
>  				if ($c && $c eq $hash) {
>  					close $fh; # break the pipe
>  					return ($url, $rev, $uuid, $gs);
> +				} else {
> +					$max{$url} ||= $gs->rev_db_max;
>  				}
>  			}
>  		}

-- 
Eric Wong

Re: [PATCH] git-svn: cache max revision in rev_db databases

[Sam Vilain]

Eric Wong wrote:
> Sam Vilain <sam@vilain.net> wrote:
>   
>> Cache the maximum revision for each rev_db URL rather than looking it
>> up each time.  This saves a lot of time when rebuilding indexes on a
>> freshly cloned repository.
>>     
>
> It looks correct, but I'm not sure how often we can even hit this
> optimization to make it worth it.
>
> Any live repositories and benchmarks you've run this on?
> (without the rev-list => log patch applied, which would be a big
> performance improvement if all log messages were under 16k).
>   
It may not make a huge difference with the current code, but it might
with a different rev_db back-end with higher connection cost, and it
seemed like an obvious and innocuous enough memoization.

Sam

[PATCH] git-svn: cache max revision in rev_db databases

[Sam Vilain]

From: Sam Vilain <sam@vilain.net>

Cache the maximum revision for each rev_db URL rather than looking it
up each time.  This saves a lot of time when rebuilding indexes on a
freshly cloned repository.

Signed-off-by: Sam Vilain <sam.vilain@catalyst.net.nz>
---
 git-svn.perl |    4 ++++
 1 files changed, 4 insertions(+), 0 deletions(-)

diff --git a/git-svn.perl b/git-svn.perl
index 556cd7d..a8b6669 100755
--- a/git-svn.perl
+++ b/git-svn.perl
@@ -801,6 +801,7 @@ sub working_head_info {
 	my ($head, $refs) = @_;
 	my ($fh, $ctx) = command_output_pipe('log', $head);
 	my $hash;
+	my %max;
 	while (<$fh>) {
 		if ( m{^commit ($::sha1)$} ) {
 			unshift @$refs, $hash if $hash and $refs;
@@ -810,11 +811,14 @@ sub working_head_info {
 		next unless s{^\s*(git-svn-id:)}{$1};
 		my ($url, $rev, $uuid) = extract_metadata($_);
 		if (defined $url && defined $rev) {
+			next if $max{$url} and $max{$url} < $rev;
 			if (my $gs = Git::SVN->find_by_url($url)) {
 				my $c = $gs->rev_db_get($rev);
 				if ($c && $c eq $hash) {
 					close $fh; # break the pipe
 					return ($url, $rev, $uuid, $gs);
+				} else {
+					$max{$url} ||= $gs->rev_db_max;
 				}
 			}
 		}
-- 
1.5.2.1.1131.g3b90

Re: [PATCH] git-svn: cache max revision in rev_db databases

[Junio C Hamano]

Sam Vilain <sam.vilain@catalyst.net.nz> writes:

> Cache the maximum revision for each rev_db URL rather than looking it
> up each time.  This saves a lot of time when rebuilding indexes on a
> freshly cloned repository.
>
> Signed-off-by: Sam Vilain <sam.vilain@catalyst.net.nz>

I think both the previous one from Sam that makes it use git-log
instead of git-rev-list and this one looks sane.  Ack/Nack is
appreciated.

> ---
>  git-svn.perl |    4 ++++
>  1 files changed, 4 insertions(+), 0 deletions(-)
>
> diff --git a/git-svn.perl b/git-svn.perl
> index 556cd7d..a8b6669 100755
> --- a/git-svn.perl
> +++ b/git-svn.perl
> @@ -801,6 +801,7 @@ sub working_head_info {
>  	my ($head, $refs) = @_;
>  	my ($fh, $ctx) = command_output_pipe('log', $head);
>  	my $hash;
> +	my %max;
>  	while (<$fh>) {
>  		if ( m{^commit ($::sha1)$} ) {
>  			unshift @$refs, $hash if $hash and $refs;
> @@ -810,11 +811,14 @@ sub working_head_info {
>  		next unless s{^\s*(git-svn-id:)}{$1};
>  		my ($url, $rev, $uuid) = extract_metadata($_);
>  		if (defined $url && defined $rev) {
> +			next if $max{$url} and $max{$url} < $rev;
>  			if (my $gs = Git::SVN->find_by_url($url)) {
>  				my $c = $gs->rev_db_get($rev);
>  				if ($c && $c eq $hash) {
>  					close $fh; # break the pipe
>  					return ($url, $rev, $uuid, $gs);
> +				} else {
> +					$max{$url} ||= $gs->rev_db_max;
>  				}
>  			}
>  		}
> -- 
> 1.5.2.1.1131.g3b90

Re: [PATCH] git-svn: cache max revision in rev_db databases

[Eric Wong]

Junio C Hamano <gitster@pobox.com> wrote:
> Sam Vilain <sam.vilain@catalyst.net.nz> writes:
> 
> > Cache the maximum revision for each rev_db URL rather than looking it
> > up each time.  This saves a lot of time when rebuilding indexes on a
> > freshly cloned repository.
> >
> > Signed-off-by: Sam Vilain <sam.vilain@catalyst.net.nz>
> 
> I think both the previous one from Sam that makes it use git-log
> instead of git-rev-list and this one looks sane.  Ack/Nack is
> appreciated.

Now that 80583c0ef61cc966c7eee79cf3623a83197e19b8 is in, both patches
are:

Acked-by: Eric Wong <normalperson@yhbt.net>

> > ---
> >  git-svn.perl |    4 ++++
> >  1 files changed, 4 insertions(+), 0 deletions(-)
> >
> > diff --git a/git-svn.perl b/git-svn.perl
> > index 556cd7d..a8b6669 100755
> > --- a/git-svn.perl
> > +++ b/git-svn.perl
> > @@ -801,6 +801,7 @@ sub working_head_info {
> >  	my ($head, $refs) = @_;
> >  	my ($fh, $ctx) = command_output_pipe('log', $head);
> >  	my $hash;
> > +	my %max;
> >  	while (<$fh>) {
> >  		if ( m{^commit ($::sha1)$} ) {
> >  			unshift @$refs, $hash if $hash and $refs;
> > @@ -810,11 +811,14 @@ sub working_head_info {
> >  		next unless s{^\s*(git-svn-id:)}{$1};
> >  		my ($url, $rev, $uuid) = extract_metadata($_);
> >  		if (defined $url && defined $rev) {
> > +			next if $max{$url} and $max{$url} < $rev;
> >  			if (my $gs = Git::SVN->find_by_url($url)) {
> >  				my $c = $gs->rev_db_get($rev);
> >  				if ($c && $c eq $hash) {
> >  					close $fh; # break the pipe
> >  					return ($url, $rev, $uuid, $gs);
> > +				} else {
> > +					$max{$url} ||= $gs->rev_db_max;
> >  				}
> >  			}
> >  		}
> > -- 
> > 1.5.2.1.1131.g3b90
> 

-- 
Eric Wong

Re: [PATCH] git-svn: cache max revision in rev_db databases

[Junio C Hamano]

Thanks.