Re: Excruciatingly slow git-svn imports

[Eric Wong <normalperson@yhbt.net>]

Avery Pennarun <apenwarr@gmail.com> wrote:
> On 5/5/08, Eric Wong <normalperson@yhbt.net> wrote:
> > Interesting.  By  "These commits seemed all to have thousands of files",
> >  you mean the first 35K that took up most of the time?  If so, yes,
> >  that's definitely a problem...
> >
> >  git-svn requests a log from SVN containing a list of all paths modified
> >  in each revision.  By default, git-svn only requests log entries for up
> >  to 100 revisions at a time to reduce memory usage.  However, having
> >  thousands of files modified for each revision would still be
> >  problematic, as would having insanely long commit messages.
> 
> On my system, any branch that was created using "svn cp" of a toplevel
> directory seems to cause git-svn to (rather slowly) download every
> single file in the entire branch for the first commit on that branch,
> giving a symptom that sounds a lot like the above "commits with
> thousands of files".  I assumed this was just an intentional design
> decision in git-svn, to be slow and safe instead of fast and loose.
> Is it actually supposed to do something smarter than that?

When using "svn cp" on a top-level directory, it *should*
just show up as a single file change in the log entry.
Something like:

  A /project/branch/my-new-branch (from /project/trunk:1234)

This would not take much memory at all.
However, I've also occasionally seen stuff like this:

  A /project/branch/my-new-branch
  A /project/branch/my-new-branch/file1 (from /project/trunk/file1:1234)
  A /project/branch/my-new-branch/file2 (from /project/trunk/file2:1234)
  A /project/branch/my-new-branch/file3 (from /project/trunk/file3:1234)
  .... many more files and directories along the same lines ...

This is what I suspect Geert is seeing in his repository and causing
problems.  Perhaps something caused by cvs2svn importing those tags into
SVN originally?


But the symptom you're seeing with git-svn downloading every file seems
to be the result of using a pre-1.4.3 version of the Perl SVN bindings
which lacked a working do_switch() function.  I fallback to using
do_update() and checking out a new tree for SVN 1.4.2 and before.
So yes, I'm definitely safe, slow and _lazy_ by falling back to
do_update() instead of doing something fancy to workaround something
that's already fixed in SVN :)

-- 
Eric Wong