Re: Anomaly with the new code - Re: git-svn performance

[Hin-Tak Leung <htl10@users.sourceforge.net>]

------------------------------
On Mon, Oct 27, 2014 06:38 GMT Eric Wong wrote:

>Which SVN version are you using?  I'm cloning (currently on r373xx)
>https://svn.r-project.org/R using --stdlayout and
>unable to see memory growth of the git-svn Perl process beyond 40M
>(on a 32-bit system).
>
>I also tried http:// (not https), svn+ssh:// on my local (64-bit) system
>and did not see memory growth, either:
>
>    http://mid.gmane.org/20141027014033.GA4189@dcvr.yhbt.net
>
>I'm using svn 1.6.17 on Debian stable in all cases.

The memory consumption does seem to go up a good deal after r48xxx -ish (the total
being about 67xxx-ish now), when there are a fair number of branches. Seeing as
you seem to be able to make the memory consumption drops further,
I'll rebuild git with dropping/adding those patches now.

I also just realised "/usr/bin/time -v git svn fetch --all" also includes the periodic auto-
garbage collection from git itself if fetching more than a number of commits,
so may not be accurate once git svn's memory consumption drops below
a certain level. Is there any way of coping with that?

I made a 3rd clone yesterday - it took 8 hours 15 minutes, and 
	Command being timed: "git svn fetch --all"
	User time (seconds): 6897.80
	System time (seconds): 18853.08
	Percent of CPU this job got: 86%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 8:14:00
...
	Maximum resident set size (kbytes): 675436

and fetching the next 8 commits:

$ /usr/bin/time -v git svn fetch --all
	M	doc/NEWS.Rd
r66871 = 0a7f50fc04dee174229513a0d80fecfcd12975ca (refs/remotes/trunk)
...
	M	doc/manual/R-exts.texi
r66879 = ede68f65df714c3ba283579d85105393c1eccc80 (refs/remotes/trunk)
Auto packing the repository in background for optimum performance.
See "git help gc" for manual housekeeping.
	Command being timed: "git svn fetch --all"
	User time (seconds): 856.82
	System time (seconds): 29.78
	Percent of CPU this job got: 98%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 15:03.39
...
	Maximum resident set size (kbytes): 791088

and quite similar against the 2nd clone, but against the first clone (which were created
by fetching every few days over a few years):

	Command being timed: "git svn fetch --all"
	User time (seconds): 518.00
	System time (seconds): 28.62
	Percent of CPU this job got: 98%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 9:16.84
...
	Maximum resident set size (kbytes): 403160

So it seems the first clone is rather different from the recent ones. I haven't got round to compare
the branches yet - it is actually easier than I thought, since I only need to compare
the branch HEADs. (I already mentioned that trunk is different, due to a blank vs 3 word
commit message about 2 years ago - I reckon I might see similar issues in the other branches
- I'll go and write a script to check that now).

All recent fetch were done with git 2.1.0 patched with the 6 patches I mentioned, on fedora 20
x86_64.

BTW, I have been meaning to ask - are you the same "Eric Wong" who maintained
some chinese packages on Debian some years ago? :-)