"git svn mkdirs" is very slow

"git svn mkdirs" is very slow

[Michael Haggerty]

Hello,

I've noticed that in our repository, "git svn mkdirs" or any other
command that calls mkemptydirs() is very slow (approximately 8.5 s even
with a warm disk cache).  Presumably this is related to the fact that
the unhandled.log.gz for our Subversion trunk is almost 10 Mb.

Of course it would be nice if this were faster.

But we have laid out our repository to make the presence/absence of
empty directories irrelevant, so we would be just as happy if git-svn
would *not* create empty directories, like in the good old days.  So I
wanted to implement the following feature:

1. An svn.autoMkdirs / svn-remote.<name>.autoMkdirs configuration
variable.  The value should default to true for backwards compatibility.

2. Only call mkemptydirs() if this variable is set to true.

3. Make an exception for "git svn mkdirs", which should do its thing
regardless of how this configuration option is set.

I think it should only be about a 10-line change, plus documentation and
tests.  Unfortunately, my perl-foo is very limited, and it will take me
a while to figure out how option parsing and handling works in git-svn.

Would this feature be welcome?

Is there anybody willing to make the Perl changes?  I would be willing
to work on the documentation and test suite changes.

Thanks,
Michael

-- 
Michael Haggerty
mhagger@alum.mit.edu
http://softwareswirl.blogspot.com/

Re: "git svn mkdirs" is very slow

[Junio C Hamano]

Michael Haggerty <mhagger@alum.mit.edu> writes:

> So I
> wanted to implement the following feature:
>
> 1. An svn.autoMkdirs / svn-remote.<name>.autoMkdirs configuration
> variable.  The value should default to true for backwards compatibility.
>
> 2. Only call mkemptydirs() if this variable is set to true.
>
> 3. Make an exception for "git svn mkdirs", which should do its thing
> regardless of how this configuration option is set.
>
> I think it should only be about a 10-line change, plus documentation and
> tests.  Unfortunately, my perl-foo is very limited, and it will take me
> a while to figure out how option parsing and handling works in git-svn.
>
> Would this feature be welcome?
>
> Is there anybody willing to make the Perl changes?  I would be willing
> to work on the documentation and test suite changes.

Sounds like a sensible thing to do, but I wonder if we also want a command
line option --automkdirs (or --auto-create-empty-directories) in %fc_opts
to make it overridable from the command line.  Perhaps it is not worth it.

A completely untested patch is here.

 git-svn.perl |   18 ++++++++++++++++--
 1 files changed, 16 insertions(+), 2 deletions(-)

diff --git a/git-svn.perl b/git-svn.perl
index a5857c1..fe4c716 100755
--- a/git-svn.perl
+++ b/git-svn.perl
@@ -781,6 +781,16 @@ sub cmd_find_rev {
 	print "$result\n" if $result;
 }
 
+sub auto_create_empty_directories {
+	my ($gs) = @_;
+	my $var = eval { command_oneline('config', '--get', '--bool',
+					 "svn-remote.$gs->{repo_id}.automkdirs") };
+	# By default, create empty directories by consulting the unhandled log,
+	# but allow setting it to 'false' to skip it.  I wonder if the variable
+	# should be "skip create empty directories", though...
+	return !($var && $var eq 'false');
+}
+
 sub cmd_rebase {
 	command_noisy(qw/update-index --refresh/);
 	my ($url, $rev, $uuid, $gs) = working_head_info('HEAD');
@@ -804,7 +814,9 @@ sub cmd_rebase {
 		$_fetch_all ? $gs->fetch_all : $gs->fetch;
 	}
 	command_noisy(rebase_cmd(), $gs->refname);
-	$gs->mkemptydirs;
+	if (auto_create_empty_directories($gs)) {
+		$gs->mkemptydirs;
+	}
 }
 
 sub cmd_show_ignore {
@@ -1242,7 +1254,9 @@ sub post_fetch_checkout {
 	command_noisy(qw/read-tree -m -u -v HEAD HEAD/);
 	print STDERR "Checked out HEAD:\n  ",
 	             $gs->full_url, " r", $gs->last_rev, "\n";
-	$gs->mkemptydirs($gs->last_rev);
+	if (auto_create_empty_directories($gs)) {
+		$gs->mkemptydirs($gs->last_rev);
+	}
 }
 
 sub complete_svn_url {

[PATCH] git-svn: add an option to skip the creation of empty directories

[Michael Haggerty]

"git svn mkdirs" (which creates empty directories in the current
working copy) can be very slow and is often unnecessary.  Provide a
config file option "svn-remote.<name>.automkdirs" that prevents empty
directories from being created automatically.  (They are still created
if "git svn mkdirs" is invoked explicitly.)

Based-on-patch-by: Junio C Hamano <gitster@pobox.com>
Signed-off-by: Michael Haggerty <mhagger@alum.mit.edu>
---
I hope I have done the attribution correctly.  The git-svn.perl file
change is by Junio and the doc and test changes are by me.

Regarding the name of the option: I prefer "automkdirs" to something
like "skipautomkdirs" because the latter is effectively the inverse of
doing something, leading to confusing double-negatives like "if (!
skipautomkdirs)".  But I am not adamant about this preference.

For completeness it might be nice to have corresponding command-line
options like --auto-mkdirs and --no-auto-mkdirs, but I doubt that it
is worth the extra effort.  If you want to create the directories when
automkdirs==false, you can simply run "git svn mkdirs".  It is more
inconvenient if you have automkdirs==true but occasionally want to
skip the creation of the directories, but this seems an uncommon
scenario.

 Documentation/git-svn.txt     |   10 ++++++++++
 git-svn.perl                  |   17 +++++++++++++++--
 t/t9146-git-svn-empty-dirs.sh |   17 +++++++++++++++++
 3 files changed, 42 insertions(+), 2 deletions(-)

diff --git a/Documentation/git-svn.txt b/Documentation/git-svn.txt
index ea8fafd..f470fbf 100644
--- a/Documentation/git-svn.txt
+++ b/Documentation/git-svn.txt
@@ -343,6 +343,8 @@ Any other arguments are passed directly to 'git log'
 	Empty directories are automatically recreated when using
 	"git svn clone" and "git svn rebase", so "mkdirs" is intended
 	for use after commands like "git checkout" or "git reset".
+	(See the svn-remote.<name>.automkdirs config file option for
+	more information.)
 
 'commit-diff'::
 	Commits the diff of two tree-ish arguments from the
@@ -663,6 +665,14 @@ svn.pathnameencoding::
 	locales to avoid corrupted file names with non-ASCII characters.
 	Valid encodings are the ones supported by Perl's Encode module.
 
+svn-remote.<name>.automkdirs::
+	Normally, the "git svn clone" and "git svn rebase" commands
+	attempt to recreate empty directories that are in the
+	Subversion repository.  If this option is set to "false", then
+	empty directories will only be created if the "git svn mkdirs"
+	command is run explicitly.  If unset, 'git svn' assumes this
+	option to be "true".
+
 Since the noMetadata, rewriteRoot, rewriteUUID, useSvnsyncProps and useSvmProps
 options all affect the metadata generated and used by 'git svn'; they
 *must* be set in the configuration file before any history is imported
diff --git a/git-svn.perl b/git-svn.perl
index a5857c1..9a9b094 100755
--- a/git-svn.perl
+++ b/git-svn.perl
@@ -781,6 +781,15 @@ sub cmd_find_rev {
 	print "$result\n" if $result;
 }
 
+sub auto_create_empty_directories {
+	my ($gs) = @_;
+	my $var = eval { command_oneline('config', '--get', '--bool',
+					 "svn-remote.$gs->{repo_id}.automkdirs") };
+	# By default, create empty directories by consulting the unhandled log,
+	# but allow setting it to 'false' to skip it.
+	return !($var && $var eq 'false');
+}
+
 sub cmd_rebase {
 	command_noisy(qw/update-index --refresh/);
 	my ($url, $rev, $uuid, $gs) = working_head_info('HEAD');
@@ -804,7 +813,9 @@ sub cmd_rebase {
 		$_fetch_all ? $gs->fetch_all : $gs->fetch;
 	}
 	command_noisy(rebase_cmd(), $gs->refname);
-	$gs->mkemptydirs;
+	if (auto_create_empty_directories($gs)) {
+		$gs->mkemptydirs;
+	}
 }
 
 sub cmd_show_ignore {
@@ -1242,7 +1253,9 @@ sub post_fetch_checkout {
 	command_noisy(qw/read-tree -m -u -v HEAD HEAD/);
 	print STDERR "Checked out HEAD:\n  ",
 	             $gs->full_url, " r", $gs->last_rev, "\n";
-	$gs->mkemptydirs($gs->last_rev);
+	if (auto_create_empty_directories($gs)) {
+		$gs->mkemptydirs($gs->last_rev);
+	}
 }
 
 sub complete_svn_url {
diff --git a/t/t9146-git-svn-empty-dirs.sh b/t/t9146-git-svn-empty-dirs.sh
index 158c8e3..6d3130e 100755
--- a/t/t9146-git-svn-empty-dirs.sh
+++ b/t/t9146-git-svn-empty-dirs.sh
@@ -28,6 +28,23 @@ test_expect_success 'empty directories exist' '
 	)
 '
 
+test_expect_success 'option automkdirs set to false' '
+	(
+		git svn init "$svnrepo" cloned-no-mkdirs &&
+		cd cloned-no-mkdirs &&
+		git config svn-remote.svn.automkdirs false &&
+		git svn fetch &&
+		for i in a b c d d/e d/e/f "weird file name"
+		do
+			if test -d "$i"
+			then
+				echo >&2 "$i exists"
+				exit 1
+			fi
+		done
+	)
+'
+
 test_expect_success 'more emptiness' '
 	svn_cmd mkdir -m "bang bang"  "$svnrepo"/"! !"
 '
-- 
1.7.4.2