git svn's performance on cloning mono's branches/tags...

git svn's performance on cloning mono's branches/tags...

[Hin-Tak Leung]

(I'm not on list so please CC) - I am trying to git svn clone mono's
repository, and for some strange reason it is doing a complete check
out from revision 1 all the way to each branch/tag . Is this normal or
am I doing anything wrong? I have git svn clone a few other things
recently (R, ghostscript) and I don't remember git doing this, but R
is quite a normal svn layout and I think I only have ghostcript trunk
(no branch/tag).

What I did was this: I started out with just git-svn clone trunk
(which generated the first two [now-commented out] lines), then I
thought I also want 2.4 branch so I added the next commented-out
section, and ran 'git svn fetch --all'. Then I thought I like the
whole thing - so I commented out most of it and added the 4 lines as
blow - which is also my current git config.

The problem is that it seems to treat every tag as a branch, and also
seem to want to check out every revision from r1 up to r3382 for
MONO_0_10, etc.
to mono/tags/MONO_0_10@3382 , and r1 to r2598 for mono/tags/MONO_0_9@2598 , etc.

--------------------------------------------------------------------
[svn-remote "svn"]
#	url = svn://anonsvn.mono-project.com/source/trunk/mono
#	fetch = :refs/remotes/git-svn
  url = svn://anonsvn.mono-project.com/source
  tags = tags/*/mono:refs/remotes/mono/tags/*
  branches = branches/*/mono:refs/remotes/mono/branches/*
  trunk = trunk/mono:refs/remotes/git-svn
#[svn-remote "mono24"]
#	url = svn://anonsvn.mono-project.com/source/branches/mono-2.4/mono
#	fetch = :refs/remotes/git-svn-mono24
--------------------------------------------------------------------

I guess my ultimate question is, how does one modify git config to
track tags and branches, etc as well, *after* having already svn
clone'd trunk?, without burning too much bandwidth?

Re: git svn's performance on cloning mono's branches/tags...

[Andreas Ericsson]

Hin-Tak Leung wrote:
> (I'm not on list so please CC) - I am trying to git svn clone mono's
> repository, and for some strange reason it is doing a complete check
> out from revision 1 all the way to each branch/tag . Is this normal or
> am I doing anything wrong? I have git svn clone a few other things
> recently (R, ghostscript) and I don't remember git doing this, but R
> is quite a normal svn layout and I think I only have ghostcript trunk
> (no branch/tag).
> 
> What I did was this: I started out with just git-svn clone trunk
> (which generated the first two [now-commented out] lines), then I
> thought I also want 2.4 branch so I added the next commented-out
> section, and ran 'git svn fetch --all'. Then I thought I like the
> whole thing - so I commented out most of it and added the 4 lines as
> blow - which is also my current git config.
> 
> The problem is that it seems to treat every tag as a branch,

This is normal. SVN makes no real distinction between tags and branches,
so git-svn has no way of detecting which is which. SVN also allows
committing to "tags", so it doesn't make sense for git to store what
SVN calls tags as git tags.

As for your other questions, I don't know.

-- 
Andreas Ericsson                   andreas.ericsson@op5.se
OP5 AB                             www.op5.se
Tel: +46 8-230225                  Fax: +46 8-230231

Considering the successes of the wars on alcohol, poverty, drugs and
terror, I think we should give some serious thought to declaring war
on peace.

Re: git svn's performance on cloning mono's branches/tags...

[Hin-Tak Leung]

On Fri, Jun 26, 2009 at 6:15 AM, Andreas Ericsson<ae@op5.se> wrote:
> Hin-Tak Leung wrote:
>>
>> (I'm not on list so please CC) - I am trying to git svn clone mono's
>> repository, and for some strange reason it is doing a complete check
>> out from revision 1 all the way to each branch/tag . Is this normal or
>> am I doing anything wrong? I have git svn clone a few other things
>> recently (R, ghostscript) and I don't remember git doing this, but R
>> is quite a normal svn layout and I think I only have ghostcript trunk
>> (no branch/tag).
>>
>> What I did was this: I started out with just git-svn clone trunk
>> (which generated the first two [now-commented out] lines), then I
>> thought I also want 2.4 branch so I added the next commented-out
>> section, and ran 'git svn fetch --all'. Then I thought I like the
>> whole thing - so I commented out most of it and added the 4 lines as
>> blow - which is also my current git config.
>>
>> The problem is that it seems to treat every tag as a branch,
>
> This is normal. SVN makes no real distinction between tags and branches,
> so git-svn has no way of detecting which is which. SVN also allows
> committing to "tags", so it doesn't make sense for git to store what
> SVN calls tags as git tags.
>
> As for your other questions, I don't know.
>
> --
> Andreas Ericsson                   andreas.ericsson@op5.se
> OP5 AB                             www.op5.se
> Tel: +46 8-230225                  Fax: +46 8-230231
>
> Considering the successes of the wars on alcohol, poverty, drugs and
> terror, I think we should give some serious thought to declaring war
> on peace.
>

I guess my question is two-fold:
1) what is the correct/recommended way of adding tags/branches *after*
already cloning trunk?

2) the current way of cloning svn repository is quite inefficient -
for every tag/branch, it does:
r1 -> rX branch 1
r1 -> rN tag 1
r1 -> rY branch 2
r1 -> rM tag 2
r1 -> rZ branch 3
etc

but I think it is possible to track branches to the point where it
branches off (when 'svn copy' happens), rather than all the way to r1?
And in that sense tags are just very short branches.

The current behavior is quite bandwith consuming for projects with a
long history and lots of tags, like mono...

Hin-Tak

Re: git svn's performance on cloning mono's branches/tags...

[Andreas Ericsson]

Hin-Tak Leung wrote:
> On Fri, Jun 26, 2009 at 6:15 AM, Andreas Ericsson<ae@op5.se> wrote:
>> Hin-Tak Leung wrote:
>>> (I'm not on list so please CC) - I am trying to git svn clone mono's
>>> repository, and for some strange reason it is doing a complete check
>>> out from revision 1 all the way to each branch/tag . Is this normal or
>>> am I doing anything wrong? I have git svn clone a few other things
>>> recently (R, ghostscript) and I don't remember git doing this, but R
>>> is quite a normal svn layout and I think I only have ghostcript trunk
>>> (no branch/tag).
>>>
>>> What I did was this: I started out with just git-svn clone trunk
>>> (which generated the first two [now-commented out] lines), then I
>>> thought I also want 2.4 branch so I added the next commented-out
>>> section, and ran 'git svn fetch --all'. Then I thought I like the
>>> whole thing - so I commented out most of it and added the 4 lines as
>>> blow - which is also my current git config.
>>>
>>> The problem is that it seems to treat every tag as a branch,
>> This is normal. SVN makes no real distinction between tags and branches,
>> so git-svn has no way of detecting which is which. SVN also allows
>> committing to "tags", so it doesn't make sense for git to store what
>> SVN calls tags as git tags.
>>
>> As for your other questions, I don't know.
>>
>> --
>> Andreas Ericsson                   andreas.ericsson@op5.se
>> OP5 AB                             www.op5.se
>> Tel: +46 8-230225                  Fax: +46 8-230231
>>
>> Considering the successes of the wars on alcohol, poverty, drugs and
>> terror, I think we should give some serious thought to declaring war
>> on peace.
>>
> 
> I guess my question is two-fold:
> 1) what is the correct/recommended way of adding tags/branches *after*
> already cloning trunk?
> 

I don't know.

> 2) the current way of cloning svn repository is quite inefficient -
> for every tag/branch, it does:
> r1 -> rX branch 1
> r1 -> rN tag 1
> r1 -> rY branch 2
> r1 -> rM tag 2
> r1 -> rZ branch 3
> etc
> 
> but I think it is possible to track branches to the point where it
> branches off (when 'svn copy' happens), rather than all the way to r1?

This isn't really a question, but a statement with a question mark.

> And in that sense tags are just very short branches.
> 

Often, yes, but they *can* be committed to. SVN's inability to keep
immutable tags is one of the reasons I never looked at it to replace
CVS at $dayjob. But I digress..

> The current behavior is quite bandwith consuming for projects with a
> long history and lots of tags, like mono...
> 

Hmm. I've cloned many subversion repositories to git, and I've never
seen the issues you're seeing. Perhaps because I always cloned the
entire repository the first time, or because I normally do such things
over a high-capacity connection, or because I just fire it up and
forget about it until it's done.

What happens if you ignore the already-cloned svn tree and just do
a new import without trying to continue the old one? Not that I'll
actually do anything about it, as I don't have any problems with it,
but I'm curious even so ;-)

-- 
Andreas Ericsson                   andreas.ericsson@op5.se
OP5 AB                             www.op5.se
Tel: +46 8-230225                  Fax: +46 8-230231

Considering the successes of the wars on alcohol, poverty, drugs and
terror, I think we should give some serious thought to declaring war
on peace.

Re: git svn's performance on cloning mono's branches/tags...

[Hin-Tak Leung]

On Fri, Jun 26, 2009 at 11:57 AM, Andreas Ericsson<ae@op5.se> wrote:
>>> Hin-Tak Leung wrote:
>>>>
>>>> (I'm not on list so please CC)

> Hmm. I've cloned many subversion repositories to git, and I've never
> seen the issues you're seeing. Perhaps because I always cloned the
> entire repository the first time, or because I normally do such things
> over a high-capacity connection, or because I just fire it up and
> forget about it until it's done.

It is not about the capacity/throughput of the connection, but the
latency (round-trip time) of many small transactions - and most of the
unnecessary...

> What happens if you ignore the already-cloned svn tree and just do
> a new import without trying to continue the old one? Not that I'll
> actually do anything about it, as I don't have any problems with it,
> but I'm curious even so ;-)

I did wonder about that - and so I just stopped my branch-tracking
fetch and start all over with svn init -T -t and -b then svn fetch
--all. The first few hundred revisions are very promising: it is
populating the git refs for tag and branch heads
as it goes up.

There are tutorials online about modifying git/config like I did to
adding branches, etc... maybe the example section of git-svn can be
updated with a couple of sentence on 'just throw it away and start
from the beginning if you change your mind from trunk-only to
complex-layout' .

OTOH, I think I would have liked to preserve the compressed object
store, and I think it is possible to graft an object store on an empty
init? The advantage is just that the disc space usage does not widely
fluctuate & no gc steps in the middle.

Thanks for the response anyhow - I just need to remember to blow
trunk-only away and start over next time I change my mind:-).

Re: git svn's performance on cloning mono's branches/tags...

[Hin-Tak Leung]

Starting afresh with -T/-t/-b is transversing the branches and tags
once, rather that over and over starting from r1.

Here is the starting afresh config:

--------------------------
[svn-remote "svn"]
	url = svn://anonsvn.mono-project.com/source
	fetch = trunk/mono:refs/remotes/trunk
	branches = branches/*/mono/:refs/remotes/*
	tags = tags/*/mono/:refs/remotes/tags/*
--------------------------

Here is my attempt at contiuation/changing my mind after getting at trunk only:
--------------------------
[svn-remote "svn"]
#	url = svn://anonsvn.mono-project.com/source/trunk/mono
#	fetch = :refs/remotes/git-svn
  url = svn://anonsvn.mono-project.com/source
  tags = tags/*/mono:refs/remotes/mono/tags/*
  branches = branches/*/mono:refs/remotes/mono/branches/*
  trunk = trunk/mono:refs/remotes/git-svn
-------------------------
the tags, branches, url entries are effectively the same, I think - so
the main difference is using trunk instead of fetch.
Why does it make any difference? The example at the bottom of
git-svn's man page uses the trunk notation. Maybe it should be
changed?

Re: git svn's performance on cloning mono's branches/tags...

[Hin-Tak Leung]

On Sat, Jun 27, 2009 at 2:17 AM, Hin-Tak Leung<hintak.leung@gmail.com> wrote:

> the tags, branches, url entries are effectively the same, I think - so
> the main difference is using trunk instead of fetch.
> Why does it make any difference? The example at the bottom of
> git-svn's man page uses the trunk notation. Maybe it should be
> changed?
>

To answer my question - I think the man page should be updated.

Also, while searching 'git svn speed', I found that a git clone of
mono is available:
http://www.mono-project.com/GitSVN
A shame this page is not referenced my other devel resources on mono's
web site at all... also it uses a real user's account to make the
clone (rather than annonymous svn), so to update the git clone by
git-svn, one needs a real dev account - that is not a problem for the
purpose of sending in patches as the occasional dev.

Re: git svn's performance on cloning mono's branches/tags...

[Eric Wong]

Hin-Tak Leung <hintak.leung@gmail.com> wrote:
> On Sat, Jun 27, 2009 at 2:17 AM, Hin-Tak Leung<hintak.leung@gmail.com> wrote:
> 
> > the tags, branches, url entries are effectively the same, I think - so
> > the main difference is using trunk instead of fetch.
> > Why does it make any difference? The example at the bottom of
> > git-svn's man page uses the trunk notation. Maybe it should be
> > changed?
> 
> To answer my question - I think the man page should be updated.

Somebody actually fixed it a while back:

commit 0e5e69a355b7bdd1af6ca33ac7ee35299bda368e
Author: Wesley J. Landaker <wjl@icecavern.net>
Date:   Wed Apr 1 16:05:01 2009 -0600

    Documentation: git-svn: fix trunk/fetch svn-remote key typo


As far as performance goes, SVN's flexibility of tagging and having
multiple subprojects interacts quite badly with git svn's --stdlayout
behavior.

With --stdoulayout, git svn supports tags/branches that are directly
descended from the top-level of trunk:

	svn cp $root/trunk $root/tags/0.1.0

However, it looks like mono does things like this:

	svn cp $root/trunk/mono $root/tags/mono-0.1.0

And when git svn sees that $root/tags/mono-0.1.0's parent is
$root/trunk/mono and not $root/trunk (which it's actually following),
it will try to fetch the complete history of $root/trunk/mono
($root/trunk may not contain all the history $root/trunk/mono contained,
either).

-- 
Eric Wong

Re: git svn's performance on cloning mono's branches/tags...

[Hin-Tak Leung]

On Thu, Jul 2, 2009 at 9:16 AM, Eric Wong <normalperson@yhbt.net> wrote:
> Hin-Tak Leung <hintak.leung@gmail.com> wrote:
>> On Sat, Jun 27, 2009 at 2:17 AM, Hin-Tak Leung<hintak.leung@gmail.com> wrote:
>>
>> > the tags, branches, url entries are effectively the same, I think - so
>> > the main difference is using trunk instead of fetch.
>> > Why does it make any difference? The example at the bottom of
>> > git-svn's man page uses the trunk notation. Maybe it should be
>> > changed?
>>
>> To answer my question - I think the man page should be updated.
>
> Somebody actually fixed it a while back:
>
> commit 0e5e69a355b7bdd1af6ca33ac7ee35299bda368e
> Author: Wesley J. Landaker <wjl@icecavern.net>
> Date:   Wed Apr 1 16:05:01 2009 -0600
>
>    Documentation: git-svn: fix trunk/fetch svn-remote key typo
>
>
> As far as performance goes, SVN's flexibility of tagging and having
> multiple subprojects interacts quite badly with git svn's --stdlayout
> behavior.
>
> With --stdoulayout, git svn supports tags/branches that are directly
> descended from the top-level of trunk:
>
>        svn cp $root/trunk $root/tags/0.1.0
>
> However, it looks like mono does things like this:
>
>        svn cp $root/trunk/mono $root/tags/mono-0.1.0
>
> And when git svn sees that $root/tags/mono-0.1.0's parent is
> $root/trunk/mono and not $root/trunk (which it's actually following),
> it will try to fetch the complete history of $root/trunk/mono
> ($root/trunk may not contain all the history $root/trunk/mono contained,
> either).
>
> --
> Eric Wong
>


Hmm, I am having another problem with git-svn going back and download
everything over and over with this:

git svn clone -s https://ndiswrapper.svn.sourceforge.net/svnroot/ndiswrapper

I am going to do two-step init -s then fetch --all now to see if it helps.

(it is probably not entirely standard layout with the extra CVSROOT?)

Re: git svn's performance on cloning mono's branches/tags...

[Eric Wong]

Hin-Tak Leung <hintak.leung@gmail.com> wrote:
> Hmm, I am having another problem with git-svn going back and download
> everything over and over with this:
> 
> git svn clone -s https://ndiswrapper.svn.sourceforge.net/svnroot/ndiswrapper
> 
> I am going to do two-step init -s then fetch --all now to see if it helps.
> 
> (it is probably not entirely standard layout with the extra CVSROOT?)

Hi,

The ndiswrapper layout looks to be *almost* standard, the CVSROOT can
probably safely be ignored, and you should be able to _mostly_ track it
with the following config:

[svn-remote "svn"]
        url = https://ndiswrapper.svn.sourceforge.net/svnroot/ndiswrapper
        fetch = trunk/ndiswrapper:refs/remotes/trunk
        branches = branches/*/ndiswrapper:refs/remotes/*
        tags = tags/*/ndiswrapper:refs/remotes/tags/*

Notice how the glob can appear in the middle of the branches and tags
configs on the remote side: foo/*/bar

Unfortunately, some of the tags seem to be inconsistently tagged and
some had tags/TAGNAME/README whereas others had
tags/TAGNAME/ndiswrapper/README (I'm using "README" to designate where
the logical top-level working directory would be).

"svn log -v https://ndiswrapper.svn.sourceforge.net/svnroot/ndiswrapper/tags"
should help find the ones that are tagged at the wrong depth:

VERSION_1_24 for example:

	fetch = tags/VERSION_1_24:refs/remotes/tags/VERSION_1_24

-- 
Eric Wong