[PATCH] Add git svn gc command

[PATCH] Add git svn gc command

[Robert Zeh]

Add a git svn gc command that gzips all unhandled.log files, and  
removes all index files under .git/svn.

Signed-off-by: Robert Allan Zeh <robert.a.zeh@gmail.com>
---
  Documentation/git-svn.txt |    4 ++++
  git-svn.perl              |   37 +++++++++++++++++++++++++++++++++++++
  t/t9140-git-svn-gc.sh     |   45 ++++++++++++++++++++++++++++++++++++ 
+++++++++
  3 files changed, 86 insertions(+), 0 deletions(-)
  create mode 100755 t/t9140-git-svn-gc.sh

diff --git a/Documentation/git-svn.txt b/Documentation/git-svn.txt
index 10af599..6d0753e 100644
--- a/Documentation/git-svn.txt
+++ b/Documentation/git-svn.txt
@@ -348,6 +348,10 @@ Any other arguments are passed directly to 'git  
log'
         "checksum mismatch" (missed a modification).  If the problem
         file cannot be ignored forever (with --ignore-paths) the only
         way to repair the repo is to use 'reset'.
+
+'gc':: Compress $GIT_DIR/svn/<refname>/unhandled.log files in .git/svn
+    and remove $GIT_DIR/svn/<refname>index files in .git/svn.
+
  +
  Only the rev_map and refs/remotes/git-svn are changed.  Follow 'reset'
  with a 'fetch' and then 'git reset' or 'git rebase' to move local
diff --git a/git-svn.perl b/git-svn.perl
index cfade63..022efc0 100755
--- a/git-svn.perl
+++ b/git-svn.perl
@@ -35,11 +35,14 @@ push @Git::SVN::Ra::ISA, 'SVN::Ra';
  push @SVN::Git::Editor::ISA, 'SVN::Delta::Editor';
  push @SVN::Git::Fetcher::ISA, 'SVN::Delta::Editor';
  use Carp qw/croak/;
+use Compress::Zlib;
  use Digest::MD5;
  use IO::File qw//;
  use File::Basename qw/dirname basename/;
  use File::Path qw/mkpath/;
  use File::Spec;
+use File::Find;
+use File::Basename;
  use Getopt::Long qw/:config gnu_getopt no_ignore_case auto_abbrev/;
  use IPC::Open3;
  use Git;
@@ -217,6 +220,9 @@ my %cmd = (
                      "Undo fetches back to the specified SVN revision",
                      { 'revision|r=s' => \$_revision,
                        'parent|p' => \$_fetch_parent } ],
+    'gc' => [ \&cmd_gc,
+              "Compress unhandled.log files in .git/svn and remove  
index files in .git/svn",
+              {} ],
  );

  my $cmd;
@@ -1107,6 +1113,18 @@ sub cmd_reset {
         print "r$r = $c ($gs->{ref_id})\n";
  }

+sub cmd_gc
+{
+    use Cwd;
+       my $git_dir = $ENV{GIT_DIR};
+    my $svn_dir = "$git_dir/svn";
+    print getcwd;
+    print "\n";
+    find({ wanted => \&gc_directory,
+           no_chdir => 1},
+         "$git_dir/svn");
+}
+
  ########################### utility functions  
#########################

  sub rebase_cmd {
@@ -1527,6 +1545,25 @@ sub md5sum {
         return $md5->hexdigest();
  }

+sub gc_directory {
+    if (-f $_ && basename($_) eq "unhandled.log") {
+        my $out_filename = $_ . ".gz";
+        print $out_filename, "\n";
+        open my $in_fh, "<", $_ or die "Unable to open $_: $!\n";
+        binmode $in_fh;
+        my $gz = gzopen($out_filename, "wb") or die "Unable to open  
$out_filename: $!\n";
+
+        my $res;
+        while ($res = sysread($in_fh, my $str, 1024)) {
+            $gz->gzwrite($str) or die "Unable to write: $gzerrno\n";
+        }
+        unlink $_ or die "unlink $File::Find::name: $!\n";
+    } elsif (-f $_ && basename($_) eq "index") {
+        unlink $_ or die "unlink %_: $!\n";
+    }
+}
+
+
  package Git::SVN;
  use strict;
  use warnings;
diff --git a/t/t9140-git-svn-gc.sh b/t/t9140-git-svn-gc.sh
new file mode 100755
index 0000000..545b6c7
--- /dev/null
+++ b/t/t9140-git-svn-gc.sh
@@ -0,0 +1,45 @@
+#!/bin/sh
+#
+# Copyright (c) 2009 Robert Allan Zeh
+
+test_description='git svn gc basic tests'
+
+. ./lib-git-svn.sh
+
+mkdir import
+mkdir tmp
+cd import
+    cat >> test.txt <<\EOF
+Sample text for Subversion repository.
+EOF
+    svn_cmd import -m 'import for git svn' . "$svnrepo" > /dev/null
+cd ..
+test_expect_success 'checkout working copy from svn' 'svn co  
"$svnrepo" test_wc'
+test_expect_success 'set some properties to create an unhandled.log  
file' \
+    'cd test_wc &&
+         svn_cmd propset foo bar test.txt &&
+         svn_cmd commit -m "property set"
+     cd ..'
+
+test_expect_success \
+    'Setup repo and fetch' '
+     git svn init "$svnrepo" &&
+     git svn fetch'
+
+test_expect_success \
+    'make backup copy of unhandled.log' '
+     cp .git/svn/git-svn/unhandled.log tmp'
+
+test_expect_success \
+    'git svn gc runs' '
+     git svn gc'
+
+test_expect_success \
+    'git svn gc produces a valid gzip file' '
+     gunzip .git/svn/git-svn/unhandled.log.gz'
+
+test_expect_success \
+    'git svn gc does not change unhandled.log files' '
+     test_cmp .git/svn/git-svn/unhandled.log tmp/unhandled.log'
+
+test_done
--
1.6.4.rc1.dirty

Re: [PATCH] Add git svn gc command

[Eric Wong]

Robert Zeh <robert.a.zeh@gmail.com> wrote:
> Add a git svn gc command that gzips all unhandled.log files, and removes 
> all index files under .git/svn.
>
> Signed-off-by: Robert Allan Zeh <robert.a.zeh@gmail.com>
> ---
>  Documentation/git-svn.txt |    4 ++++
>  git-svn.perl              |   37 +++++++++++++++++++++++++++++++++++++
>  t/t9140-git-svn-gc.sh     |   45 ++++++++++++++++++++++++++++++++++++ 
> +++++++++
>  3 files changed, 86 insertions(+), 0 deletions(-)
>  create mode 100755 t/t9140-git-svn-gc.sh

Hi Robert,

Your mailer is mangling whitespace badly so the patch isn't applying
at all.  Make sure indents are real tabs like the rest of the code.

Some more comments below, but I think this will be a good addition
to git svn.

> --- a/git-svn.perl
> +++ b/git-svn.perl
> @@ -35,11 +35,14 @@ push @Git::SVN::Ra::ISA, 'SVN::Ra';
>  push @SVN::Git::Editor::ISA, 'SVN::Delta::Editor';
>  push @SVN::Git::Fetcher::ISA, 'SVN::Delta::Editor';
>  use Carp qw/croak/;
> +use Compress::Zlib;

I'd "require" Compress::Zlib lazilly so it's not loaded at startup.
It's not a stock component of Perl and not needed for the majority of
commands.

> +sub cmd_gc
> +{
> +    use Cwd;
> +       my $git_dir = $ENV{GIT_DIR};
> +    my $svn_dir = "$git_dir/svn";
> +    print getcwd;
> +    print "\n";

I'm not sure why you need to print getcwd there, leftover code from
testing/debugging?

> +sub gc_directory {
> +    if (-f $_ && basename($_) eq "unhandled.log") {
> +        my $out_filename = $_ . ".gz";
> +        print $out_filename, "\n";
> +        open my $in_fh, "<", $_ or die "Unable to open $_: $!\n";
> +        binmode $in_fh;
> +        my $gz = gzopen($out_filename, "wb") or die "Unable to open  
> $out_filename: $!\n";

gzopen with "ab" might be a better idea than "wb".  I'm not sure if
clobbering the existing log every time we run this command is a good
idea.

-- 
Eric Wong

Re: [PATCH] Add git svn gc command

[demerphq]

2009/7/20 Eric Wong <normalperson@yhbt.net>:
> Robert Zeh <robert.a.zeh@gmail.com> wrote:
>> Add a git svn gc command that gzips all unhandled.log files, and removes
>> all index files under .git/svn.
>>
>> Signed-off-by: Robert Allan Zeh <robert.a.zeh@gmail.com>
>> ---
>>  Documentation/git-svn.txt |    4 ++++
>>  git-svn.perl              |   37 +++++++++++++++++++++++++++++++++++++
>>  t/t9140-git-svn-gc.sh     |   45 ++++++++++++++++++++++++++++++++++++
>> +++++++++
>>  3 files changed, 86 insertions(+), 0 deletions(-)
>>  create mode 100755 t/t9140-git-svn-gc.sh
>
> Hi Robert,
>
> Your mailer is mangling whitespace badly so the patch isn't applying
> at all.  Make sure indents are real tabs like the rest of the code.
>
> Some more comments below, but I think this will be a good addition
> to git svn.
>
>> --- a/git-svn.perl
>> +++ b/git-svn.perl
>> @@ -35,11 +35,14 @@ push @Git::SVN::Ra::ISA, 'SVN::Ra';
>>  push @SVN::Git::Editor::ISA, 'SVN::Delta::Editor';
>>  push @SVN::Git::Fetcher::ISA, 'SVN::Delta::Editor';
>>  use Carp qw/croak/;
>> +use Compress::Zlib;
>
> I'd "require" Compress::Zlib lazilly so it's not loaded at startup.
> It's not a stock component of Perl and not needed for the majority of
> commands.

Actually, it has been a core component since 5.9.3

cheers,
Yves



-- 
perl -Mre=debug -e "/just|another|perl|hacker/"

Re: [PATCH] Add git svn gc command

[Eric Wong]

demerphq <demerphq@gmail.com> wrote:
> 2009/7/20 Eric Wong <normalperson@yhbt.net>:
> > Robert Zeh <robert.a.zeh@gmail.com> wrote:
> >> --- a/git-svn.perl
> >> +++ b/git-svn.perl
> >> @@ -35,11 +35,14 @@ push @Git::SVN::Ra::ISA, 'SVN::Ra';
> >>  push @SVN::Git::Editor::ISA, 'SVN::Delta::Editor';
> >>  push @SVN::Git::Fetcher::ISA, 'SVN::Delta::Editor';
> >>  use Carp qw/croak/;
> >> +use Compress::Zlib;
> >
> > I'd "require" Compress::Zlib lazilly so it's not loaded at startup.
> > It's not a stock component of Perl and not needed for the majority of
> > commands.
> 
> Actually, it has been a core component since 5.9.3

Ah thanks for pointing that out, I didn't notice my 5.10.x install had
it.  Nevertheless, git svn needs to continue supporting 5.8.x for a
while longer.

-- 
Eric Wong

Re: [PATCH] Add git svn gc command

[demerphq]

2009/7/20 Eric Wong <normalperson@yhbt.net>:
> demerphq <demerphq@gmail.com> wrote:
>> 2009/7/20 Eric Wong <normalperson@yhbt.net>:
>> > Robert Zeh <robert.a.zeh@gmail.com> wrote:
>> >> --- a/git-svn.perl
>> >> +++ b/git-svn.perl
>> >> @@ -35,11 +35,14 @@ push @Git::SVN::Ra::ISA, 'SVN::Ra';
>> >>  push @SVN::Git::Editor::ISA, 'SVN::Delta::Editor';
>> >>  push @SVN::Git::Fetcher::ISA, 'SVN::Delta::Editor';
>> >>  use Carp qw/croak/;
>> >> +use Compress::Zlib;
>> >
>> > I'd "require" Compress::Zlib lazilly so it's not loaded at startup.
>> > It's not a stock component of Perl and not needed for the majority of
>> > commands.
>>
>> Actually, it has been a core component since 5.9.3
>
> Ah thanks for pointing that out, I didn't notice my 5.10.x install had
> it.  Nevertheless, git svn needs to continue supporting 5.8.x for a
> while longer.

I guess something like:

my $can_compress= eval "require Compress::Zlib; 1";

would be the right solution.




-- 
perl -Mre=debug -e "/just|another|perl|hacker/"

Re: [PATCH] Add git svn gc command

[Thomas Rast]

demerphq wrote:
> 
> my $can_compress= eval "require Compress::Zlib; 1";

Wouldn't

  my $can_compress = eval { require Compress::Zlib; 1 };

be better on the grounds that this will catch compilation errors in
the eval'd statement at compilation (of the main program) time?

-- 
Thomas Rast
trast@{inf,student}.ethz.ch

Re: [PATCH] Add git svn gc command

[demerphq]

2009/7/20 Thomas Rast <trast@student.ethz.ch>:
> demerphq wrote:
>>
>> my $can_compress= eval "require Compress::Zlib; 1";
>
> Wouldn't
>
>  my $can_compress = eval { require Compress::Zlib; 1 };
>
> be better on the grounds that this will catch compilation errors in
> the eval'd statement at compilation (of the main program) time?

Yeah, thats a good catch. In this case there is no need for the eval
STRING usage.

cheers,
Yves

-- 
perl -Mre=debug -e "/just|another|perl|hacker/"

Re: [PATCH] Add git svn gc command

[Eric Wong]

Robert Zeh <robert.a.zeh@gmail.com> wrote:
> Hi Eric,
>
> Thanks for all of the suggestions.  I've attached a patch that takes all 
> of the list suggestions into account.  I hope I've implemented the  
> can_compress changes properly.

Hi Robert, that looks fine to me.

> I believe I've fixed the white space issues; I tried a round trip and my 
> tabs were preserved.  The getcwd stuff was indeed leftover code from 
> testing; I found another left over debug statement.

The tabs seem alright, but spaces are still mangled.  Perhaps
Documentation/SubmittingPatches has tips for your mailer? (or check out
git send-email)

> I have one remaining question ---  does git svn gc need to avoid  
> removing index files if --no-metadata was set?  If it does, I'd  
> appreciate a tip on how to see if it was set.

Nope, not at all, index files are very safely removable unless there's a
clone/fetch actively running.  The rev_map files on the other hand are
not removable iff --no-metadata or --use-svm-props is used.

-- 
Eric Wong