importing multi-project svn repositories

importing multi-project svn repositories

[David Hanson]

What's the recommended way to import the full history from an svn  
repository organized as a group of projects? E.g., as described at  
http://svnbook.red-bean.com/nightly/en/ 
svn.reposadmin.planning.html#svn.reposadmin.projects.chooselayout:

/
    calc/
       trunk/
       tags/
       branches/
    calendar/
       trunk/
       tags/
       branches/
    spreadsheet/
       trunk/
       tags/
       branches/
    …

I'd like to import calc at the top level, put calc/tags/foo in git's  
tags/calc/foo and calc/branches/baz in git's heads/calc/baz. Ditto  
for calendar, spreadsheet, etc.

I suspect there's a way to use git-svnimport repeatedly with  
appropriate editing of .git/svn2git.
thanks,
dave h

Re: importing multi-project svn repositories

[Steven Grimm]

David Hanson wrote:
> I'd like to import calc at the top level, put calc/tags/foo in git's 
> tags/calc/foo and calc/branches/baz in git's heads/calc/baz. Ditto for 
> calendar, spreadsheet, etc.

Try git-svn rather than git-svnimport. The latter, AFAIK, is no longer 
really maintained. The former will do what you want pretty easily. You 
just run it like

git svn clone --prefix=calc/ -T trunk -t tags -b branches \
    http://svn/repo/url/calc gitrepo

and it'll create a git repository called "gitrepo" with the calc 
project's trunk/tags/branches as git tags. Then do

git svn clone --prefix=spreadsheet/ -T trunk -t tags -b branches \
    http://svn/repo/url/spreadsheet gitrepo

and it will add the spreadsheet branches to the same repo. The 
"--prefix" option is needed to keep the branch namespaces from 
overlapping. git-svn will make an attempt to figure out the correct 
history of the branches and tags relative to the trunk so they look like 
sane git branches.

If you want to be able to check out calc and spreadsheet simultaneously, 
then you probably need to use git's submodule support, which is kind of 
a work in progress at the moment.

-Steve

Re: importing multi-project svn repositories

[Dave Hanson]

On 5/7/07, Steven Grimm <koreth@midwinter.com> wrote:
> David Hanson wrote:
> > I'd like to import calc at the top level, put calc/tags/foo in git's
> > tags/calc/foo and calc/branches/baz in git's heads/calc/baz. Ditto for
> > calendar, spreadsheet, etc.
>
> Try git-svn rather than git-svnimport. The latter, AFAIK, is no longer
> really maintained. ...

Thanks for the tip. I was running git 1.5.0, which didn't support
"clone" to git-svn. I upgraded to 1.5.1.3 and followed your
suggestions.

If I decide to flush the svn repository in favor of git, I'll probably
do some manual rearrangement so that commands like "git tag -l" gives
useful results.
thanks,
dave h

Deprecate git-svnimport?

[Steven Grimm]

On 5/7/07, Steven Grimm <koreth@midwinter.com> wrote:
> Try git-svn rather than git-svnimport. The latter, AFAIK, is no longer
> really maintained. ...

I said that because in the time I've been using git and reading this 
list, I think I've seen a grand total of one patch to git-svnimport, a 
ton of them to git-svn, and an occasional new user posting about 
git-svnimport failing to do something that git-svn would have done if 
they'd used it instead.

It looks like git-svnimport hasn't been updated to the 1.5.x remote 
branch naming convention (it still calls the svn head tracking branch 
"origin" if its manpage is to be believed).

Originally it looked like git-svn was what you used if you wanted to 
track just one svn branch bidirectionally, and git-svnimport was how you 
tracked multiple svn branches read-only. But now that git-svn has good 
handling of multi-branch svn repositories, I wonder if git-svnimport 
still serves a purpose. Should it perhaps be deprecated and removed in a 
future release? Or is it still useful for things that git-svn doesn't 
handle?

-Steve

minimize_url in git-svn?

[Junio C Hamano]

I was trying to run git-svn against this:

	https://repo.socialtext.net:8999/svn/socialtext/trunk

This is an open source project [*1*] and the trunk is supposed
to be readable by everybody, but it seems that anything outside
that area needs authentication.  If I mimick the example in
git-svn.txt manual page to clone from there, it creates trunk,
trunk/.git, and then asks for password:

	$ URL=https://repo.socialtext.net:8999/svn/socialtext/trunk
	$ git-svn clone $URL
        Authentication realm: <https://repo.socialtext.net:8999> Auth for SVN
	Password for 'junio': ^C

I've narrowed it down to this part of git-svn.  If I tell it not
to bother "minimiz"ing the URL, it seems to import without
stepping outside of the URL it was given.

--- a/git-svn.perl
+++ b/git-svn.perl
@@ -1038,7 +1038,8 @@
 		}
 		$self->{repo_id} = $existing;
 	} else {
-		my $min_url = Git::SVN::Ra->new($url)->minimize_url;
+		my $ra = Git::SVN::Ra->new($url);
+		my $min_url = $url; # $ra->minimize_url;
 		$existing = find_existing_remote($min_url, $r);
 		if ($existing) {
 			unless ($no_write) {

Two and half questions.

 * What does minimize do, and why is it necessary?

 * The resulting git-svn remote tracking branch (and 'master')
   seems to check out fine, but I do not know what damage the
   hack to avoid minimizing is causing.  Are there any?  I see
   many 0{40} lines in trunk/.git/svn/git-svn/.rev_db.* file,
   and also many lines in unhandled.log file (+empty_dir,
   +file_prop, and +dir_prop).  Are these something to worry
   about?

 * Assuming there aren't any damage, or maybe some damage that
   would cause minor decreased functionality/interoperability,
   would it perhaps make sense to optionally allow skipping the
   minimizing to avoid this problem?  Would it make sense, or is
   the setting at socialtext site too esoteric and it isn't
   worth to worry about?


[Footnote]

*1* http://www.socialtext.net/stoss/index.cgi?developing_with_a_dev_env

Re: minimize_url in git-svn?

[Johannes Schindelin]

Hi,

On Tue, 8 May 2007, Junio C Hamano wrote:

> I was trying to run git-svn against this:
> 
> 	https://repo.socialtext.net:8999/svn/socialtext/trunk
> 
> This is an open source project [*1*] and the trunk is supposed
> to be readable by everybody, but it seems that anything outside
> that area needs authentication.  If I mimick the example in
> git-svn.txt manual page to clone from there, it creates trunk,
> trunk/.git, and then asks for password:
> 
> 	$ URL=https://repo.socialtext.net:8999/svn/socialtext/trunk
> 	$ git-svn clone $URL
>         Authentication realm: <https://repo.socialtext.net:8999> Auth for SVN
> 	Password for 'junio': ^C

FWIW I encountered the same problem with tailor.

Ciao,
Dscho

Re: minimize_url in git-svn?

[Eric Wong]

Junio C Hamano <junkio@cox.net> wrote:
> I was trying to run git-svn against this:
> 
> 	https://repo.socialtext.net:8999/svn/socialtext/trunk
> 
> This is an open source project [*1*] and the trunk is supposed
> to be readable by everybody, but it seems that anything outside
> that area needs authentication.  If I mimick the example in
> git-svn.txt manual page to clone from there, it creates trunk,
> trunk/.git, and then asks for password:
> 
> 	$ URL=https://repo.socialtext.net:8999/svn/socialtext/trunk
> 	$ git-svn clone $URL
>         Authentication realm: <https://repo.socialtext.net:8999> Auth for SVN
> 	Password for 'junio': ^C
> 
> I've narrowed it down to this part of git-svn.  If I tell it not
> to bother "minimiz"ing the URL, it seems to import without
> stepping outside of the URL it was given.
> 
> --- a/git-svn.perl
> +++ b/git-svn.perl
> @@ -1038,7 +1038,8 @@
>  		}
>  		$self->{repo_id} = $existing;
>  	} else {
> -		my $min_url = Git::SVN::Ra->new($url)->minimize_url;
> +		my $ra = Git::SVN::Ra->new($url);
> +		my $min_url = $url; # $ra->minimize_url;
>  		$existing = find_existing_remote($min_url, $r);
>  		if ($existing) {
>  			unless ($no_write) {

That should be fine.

> Two and half questions.
> 
>  * What does minimize do, and why is it necessary?

I try to connect to the root (or closer to the root) of the repository.
This allows branches and tags to be tracked more effectively without
needing reconnects.  There's a reparent function in SVN 1.4, but it
doesn't work correctly with svn:// repos last I checked (1.4.3)

>  * The resulting git-svn remote tracking branch (and 'master')
>    seems to check out fine, but I do not know what damage the
>    hack to avoid minimizing is causing.  Are there any?  I see
>    many 0{40} lines in trunk/.git/svn/git-svn/.rev_db.* file,
>    and also many lines in unhandled.log file (+empty_dir,
>    +file_prop, and +dir_prop).  Are these something to worry
>    about?

Nope.  unhandled.log is strictly informational.

.rev_db is offset-based database.  Revision numbers to git commits can
be looked up using (SVN revision * 41).

If the project has really high revision numbers (like gcc) or lots of
tags, it's a space-killer.  I've been meaning to add an optional SQLite
alternative to .rev_db for people tracking those projects.  Patches
welcome :)

>  * Assuming there aren't any damage, or maybe some damage that
>    would cause minor decreased functionality/interoperability,
>    would it perhaps make sense to optionally allow skipping the
>    minimizing to avoid this problem?  Would it make sense, or is
>    the setting at socialtext site too esoteric and it isn't
>    worth to worry about?

It *should* be automatically detecting the highest level up it can
access and stop there.  In your case, there's obviously something
broken in my code :(

I've definitely tested this as working against Seth Falcon's hedgehog
repo (URL is somewhere in the archives).  I also setup a test repository
somewhere that I can double-check against.

> [Footnote]
> 
> *1* http://www.socialtext.net/stoss/index.cgi?developing_with_a_dev_env

I'll try to take a look at that in the next few days.  I also have
segfaults to fix that I haven't gotten to :(

-- 
Eric Wong

Re: minimize_url in git-svn?

[Junio C Hamano]

Eric Wong <normalperson@yhbt.net> writes:

> .rev_db is offset-based database.  Revision numbers to git commits can
> be looked up using (SVN revision * 41).

Ah, that explains why the file looks so sparse.  Their trunk/
seems to be somewhat inactive and activities elsewhere in the
branch/ namespace we cannot view may advance SVN revisions.

>>  * Assuming there aren't any damage, or maybe some damage that
>>    would cause minor decreased functionality/interoperability,
>>    would it perhaps make sense to optionally allow skipping the
>>    minimizing to avoid this problem?  Would it make sense, or is
>>    the setting at socialtext site too esoteric and it isn't
>>    worth to worry about?
>
> It *should* be automatically detecting the highest level up it can
> access and stop there.  In your case, there's obviously something
> broken in my code :(

Perhaps, but if the user has the proper user identity to access
the full site, I suspect you would be better off getting the
credential and connect to the higher level.  In other words, the
definition of "the highest level up it can access" depends on
the user and what the user is trying to do (i.e. a user who does
have an account with that project may not be interested in
getting tags/ nor branches/, in which case the user may want to
be able to say "I am not interested in uplevel" upfront, so that
the program does not even have to ask for the password.  On the
other hand the user may want to access tags/ and branches/ in
which case the user would appreciate the current behaviour,
asking for the password and go up after authenticated).

> I've definitely tested this as working against Seth Falcon's hedgehog
> repo (URL is somewhere in the archives).  I also setup a test repository
> somewhere that I can double-check against.
>
>> [Footnote]
>> 
>> *1* http://www.socialtext.net/stoss/index.cgi?developing_with_a_dev_env
>
> I'll try to take a look at that in the next few days.  I also have
> segfaults to fix that I haven't gotten to :(

Thanks.  If I type <Return> instead of ^C in the transcript at
the beginning of my message, I get a segfault.  It may or may
not be the same one you have been chasing.

[PATCH] git-svn: don't attempt to minimize URLs by default

[Eric Wong]

For tracking branches and tags, git-svn prefers to connect
to the root of the repository or at least the level that
houses branches and tags as well as trunk.  However, users
that are accustomed to tracking a single directory have
no use for this feature.

As pointed out by Junio, users may not have permissions to
connect to connect to a higher-level path in the repository.

While the current minimize_url() function detects lack of
permissions to certain paths _after_ successful logins, it
cannot effectively determine if it is trying to access a
login-only portion of a repo when the user expects to
connect to a part where anonymous access is allowed.

For people used to the git-svnimport switches of
--trunk, --tags, --branches, they'll already pass the
repository root (or root+subdirectory), so minimize URL
isn't of too much use to them, either.

For people *not* used to git-svnimport, git-svn also
supports:

 git svn init --minimize-url \
  --trunk http://repository-root/foo/trunk \
  --branches http://repository-root/foo/branches \
  --tags http://repository-root/foo/tags

And this is where the new --minimize-url command-line switch
comes in to allow for this behavior to continue working.
---
 git-svn.perl                         |    5 +++--
 t/t9100-git-svn-basic.sh             |    2 +-
 t/t9104-git-svn-follow-parent.sh     |   13 +++++++------
 t/t9105-git-svn-commit-diff.sh       |    2 +-
 t/t9110-git-svn-use-svm-props.sh     |    7 ++++---
 t/t9111-git-svn-use-svnsync-props.sh |    6 +++---
 6 files changed, 19 insertions(+), 16 deletions(-)

diff --git a/git-svn.perl b/git-svn.perl
index 5352470..e38811a 100755
--- a/git-svn.perl
+++ b/git-svn.perl
@@ -80,6 +80,7 @@ my %icv;
 my %init_opts = ( 'template=s' => \$_template, 'shared:s' => \$_shared,
                   'trunk|T=s' => \$_trunk, 'tags|t=s' => \$_tags,
                   'branches|b=s' => \$_branches, 'prefix=s' => \$_prefix,
+                  'minimize-url|m' => \$Git::SVN::_minimize_url,
 		  'no-metadata' => sub { $icv{noMetadata} = 1 },
 		  'use-svm-props' => sub { $icv{useSvmProps} = 1 },
 		  'use-svnsync-props' => sub { $icv{useSvnsyncProps} = 1 },
@@ -820,7 +821,7 @@ use strict;
 use warnings;
 use vars qw/$default_repo_id $default_ref_id $_no_metadata $_follow_parent
             $_repack $_repack_flags $_use_svm_props $_head
-            $_use_svnsync_props $no_reuse_existing/;
+            $_use_svnsync_props $no_reuse_existing $_minimize_url/;
 use Carp qw/croak/;
 use File::Path qw/mkpath/;
 use File::Copy qw/copy/;
@@ -1037,7 +1038,7 @@ sub init_remote_config {
 				     "[svn-remote \"$existing\"]\n";
 		}
 		$self->{repo_id} = $existing;
-	} else {
+	} elsif ($_minimize_url) {
 		my $min_url = Git::SVN::Ra->new($url)->minimize_url;
 		$existing = find_existing_remote($min_url, $r);
 		if ($existing) {
diff --git a/t/t9100-git-svn-basic.sh b/t/t9100-git-svn-basic.sh
index eb628fe..70c3669 100755
--- a/t/t9100-git-svn-basic.sh
+++ b/t/t9100-git-svn-basic.sh
@@ -229,7 +229,7 @@ test_expect_failure 'exit if init-ing a would clobber a URL' "
 
 test_expect_success \
   'init allows us to connect to another directory in the same repo' "
-        git-svn init -i bar $svnrepo/bar &&
+        git-svn init --minimize-url -i bar $svnrepo/bar &&
         git config --get svn-remote.svn.fetch \
                               '^bar:refs/remotes/bar$' &&
         git config --get svn-remote.svn.fetch \
diff --git a/t/t9104-git-svn-follow-parent.sh b/t/t9104-git-svn-follow-parent.sh
index bd4f366..35aa45c 100755
--- a/t/t9104-git-svn-follow-parent.sh
+++ b/t/t9104-git-svn-follow-parent.sh
@@ -28,7 +28,7 @@ test_expect_success 'initialize repo' "
 	"
 
 test_expect_success 'init and fetch a moved directory' "
-	git-svn init -i thunk $svnrepo/thunk &&
+	git-svn init --minimize-url -i thunk $svnrepo/thunk &&
 	git-svn fetch -i thunk &&
 	test \"\`git-rev-parse --verify refs/remotes/thunk@2\`\" \
            = \"\`git-rev-parse --verify refs/remotes/thunk~1\`\" &&
@@ -68,7 +68,8 @@ test_expect_success 'follow larger parent' "
         echo hi > import/trunk/thunk/bump/thud/file &&
         svn import -m 'import a larger parent' import $svnrepo/larger-parent &&
         svn cp -m 'hi' $svnrepo/larger-parent $svnrepo/another-larger &&
-        git-svn init -i larger $svnrepo/another-larger/trunk/thunk/bump/thud &&
+        git-svn init --minimize-url -i larger \
+          $svnrepo/another-larger/trunk/thunk/bump/thud &&
         git-svn fetch -i larger &&
         git-rev-parse --verify refs/remotes/larger &&
         git-rev-parse --verify \
@@ -90,14 +91,14 @@ test_expect_success 'follow higher-level parent' "
                 cd ..
         svn mkdir -m 'new glob at top level' $svnrepo/glob &&
         svn mv -m 'move blob down a level' $svnrepo/blob $svnrepo/glob/blob &&
-        git-svn init -i blob $svnrepo/glob/blob &&
+        git-svn init --minimize-url -i blob $svnrepo/glob/blob &&
         git-svn fetch -i blob
         "
 
 test_expect_success 'follow deleted directory' "
 	svn mv -m 'bye!' $svnrepo/glob/blob/hi $svnrepo/glob/blob/bye &&
 	svn rm -m 'remove glob' $svnrepo/glob &&
-	git-svn init -i glob $svnrepo/glob &&
+	git-svn init --minimize-url -i glob $svnrepo/glob &&
 	git-svn fetch -i glob &&
 	test \"\`git cat-file blob refs/remotes/glob:blob/bye\`\" = hi &&
 	test \"\`git ls-tree refs/remotes/glob | wc -l \`\" -eq 1
@@ -127,7 +128,7 @@ test_expect_success 'follow-parent avoids deleting relevant info' "
 	  poke native/t/c.t &&
 	  svn commit -m 'reorg test' &&
 	cd .. &&
-	git-svn init -i r9270-t \
+	git-svn init --minimize-url -i r9270-t \
 	  $svnrepo/r9270/trunk/subversion/bindings/swig/perl/native/t &&
 	git-svn fetch -i r9270-t &&
 	test \`git rev-list r9270-t | wc -l\` -eq 2 &&
@@ -137,7 +138,7 @@ test_expect_success 'follow-parent avoids deleting relevant info' "
 
 test_expect_success "track initial change if it was only made to parent" "
 	svn cp -m 'wheee!' $svnrepo/r9270/trunk $svnrepo/r9270/drunk &&
-	git-svn init -i r9270-d \
+	git-svn init --minimize-url -i r9270-d \
 	  $svnrepo/r9270/drunk/subversion/bindings/swig/perl/native/t &&
 	git-svn fetch -i r9270-d &&
 	test \`git rev-list r9270-d | wc -l\` -eq 3 &&
diff --git a/t/t9105-git-svn-commit-diff.sh b/t/t9105-git-svn-commit-diff.sh
index c668dd1..318e172 100755
--- a/t/t9105-git-svn-commit-diff.sh
+++ b/t/t9105-git-svn-commit-diff.sh
@@ -33,7 +33,7 @@ test_expect_success 'test the commit-diff command' "
 
 test_expect_success 'commit-diff to a sub-directory (with git-svn config)' "
 	svn import -m 'sub-directory' import $svnrepo/subdir &&
-	git-svn init $svnrepo/subdir &&
+	git-svn init --minimize-url $svnrepo/subdir &&
 	git-svn fetch &&
 	git-svn commit-diff -r3 '$prev' '$head' &&
 	svn cat $svnrepo/subdir/readme > readme.2 &&
diff --git a/t/t9110-git-svn-use-svm-props.sh b/t/t9110-git-svn-use-svm-props.sh
index 9db0d8f..59e17f2 100755
--- a/t/t9110-git-svn-use-svm-props.sh
+++ b/t/t9110-git-svn-use-svm-props.sh
@@ -9,9 +9,10 @@ test_description='git-svn useSvmProps test'
 
 test_expect_success 'load svm repo' "
 	svnadmin load -q $rawsvnrepo < ../t9110/svm.dump &&
-	git-svn init -R arr -i bar $svnrepo/mirror/arr &&
-	git-svn init -R argh -i dir $svnrepo/mirror/argh &&
-	git-svn init -R argh -i e $svnrepo/mirror/argh/a/b/c/d/e &&
+	git-svn init --minimize-url -R arr -i bar $svnrepo/mirror/arr &&
+	git-svn init --minimize-url -R argh -i dir $svnrepo/mirror/argh &&
+	git-svn init --minimize-url -R argh -i e \
+	  $svnrepo/mirror/argh/a/b/c/d/e &&
 	git-config svn.useSvmProps true &&
 	git-svn fetch --all
 	"
diff --git a/t/t9111-git-svn-use-svnsync-props.sh b/t/t9111-git-svn-use-svnsync-props.sh
index 483d7f8..e523214 100755
--- a/t/t9111-git-svn-use-svnsync-props.sh
+++ b/t/t9111-git-svn-use-svnsync-props.sh
@@ -9,9 +9,9 @@ test_description='git-svn useSvnsyncProps test'
 
 test_expect_success 'load svnsync repo' "
 	svnadmin load -q $rawsvnrepo < ../t9111/svnsync.dump &&
-	git-svn init -R arr -i bar $svnrepo/bar &&
-	git-svn init -R argh -i dir $svnrepo/dir &&
-	git-svn init -R argh -i e $svnrepo/dir/a/b/c/d/e &&
+	git-svn init --minimize-url -R arr -i bar $svnrepo/bar &&
+	git-svn init --minimize-url -R argh -i dir $svnrepo/dir &&
+	git-svn init --minimize-url -R argh -i e $svnrepo/dir/a/b/c/d/e &&
 	git-config svn.useSvnsyncProps true &&
 	git-svn fetch --all
 	"
-- 
Eric Wong