Efficiency and correctness patches for git-svn mergeinfo support

Efficiency and correctness patches for git-svn mergeinfo support

[Sam Vilain]

This series implements some efficiency enhancements; particularly in
dealing with repositories which have a single trunk branch which
receives many merges of feature branches and/or cherry-picks.  It also
fixes a number of corner cases in the merge conversion code.

It would be nice if the people who have experienced slow git-svn
performance in these situations could test that this fixes the
performance issues, and that the resulting repositories seem to have
correct contents.

Eric, I'm using Alex's trick of a single commit which adds failing
tests - marked as _expect_failure - and then marked them as succeeding
as the series progresses.

[PATCH 1/5] git-svn: expand the svn mergeinfo test suite, highlighting some failures

[Sam Vilain]

As shown, git-svn has some problems; not all svn merges are correctly
detected, and cherry picks may incorrectly be detected as real merges.
These test cases will be marked as _success once the relevant fixes are in.

Signed-off-by: Sam Vilain <sam@vilain.net>
---
 t/t9151-svn-mergeinfo.sh   |   27 ++-
 t/t9151/make-svnmerge-dump |  166 ++++++---
 t/t9151/svn-mergeinfo.dump |  839 ++++++++++++++++++++++++++++++++++++++------
 3 files changed, 860 insertions(+), 172 deletions(-)

diff --git a/t/t9151-svn-mergeinfo.sh b/t/t9151-svn-mergeinfo.sh
index f57daf4..dc3478f 100755
--- a/t/t9151-svn-mergeinfo.sh
+++ b/t/t9151-svn-mergeinfo.sh
@@ -15,12 +15,27 @@ test_expect_success 'load svn dump' "
 	git svn fetch --all
 	"
 
-test_expect_success 'represent svn merges without intervening commits' "
-	[ `git cat-file commit HEAD^1 | grep parent | wc -l` -eq 2 ]
-	"
+test_expect_failure 'all svn merges became git merge commits' '
+	unmarked=$(git rev-list --parents --all --grep=Merge |
+		grep -v " .* " | cut -f1 -d" ")
+	[ -z "$unmarked" ]
+	'
 
-test_expect_success 'represent svn merges with intervening commits' "
-	[ `git cat-file commit HEAD | grep parent | wc -l` -eq 2 ]
-	"
+test_expect_failure 'cherry picks did not become git merge commits' '
+	bad_cherries=$(git rev-list --parents --all --grep=Cherry |
+		grep " .* " | cut -f1 -d" ")
+	[ -z "$bad_cherries" ]
+	'
+
+test_expect_success 'svn non-merge merge commits did not become git merge commits' '
+	bad_non_merges=$(git rev-list --parents --all --grep=non-merge |
+		grep " .* " | cut -f1 -d" ")
+	[ -z "$bad_non_merges" ]
+	'
+
+test_expect_failure 'everything got merged in the end' '
+	unmerged=$(git rev-list --all --not master)
+	[ -z "$unmerged" ]
+	'
 
 test_done
diff --git a/t/t9151/make-svnmerge-dump b/t/t9151/make-svnmerge-dump
index 7e3da75..d917717 100644
--- a/t/t9151/make-svnmerge-dump
+++ b/t/t9151/make-svnmerge-dump
@@ -11,93 +11,151 @@ mkdir foo.svn
 svnadmin create foo.svn
 svn co file://`pwd`/foo.svn foo
 
+commit() {
+    i=$(( $1 + 1 ))
+    shift;
+    svn commit -m "(r$i) $*" >/dev/null || exit 1
+    echo $i
+}
+
+say() {
+    echo "^[[1m * $*^[[0m"
+}
+
+i=0
 cd foo
 mkdir trunk
 mkdir branches
 svn add trunk branches
-svn commit -m "Setup trunk and branches"
-cd trunk
+i=$(commit $i "Setup trunk and branches")
 
-git cat-file blob 6683463e:Makefile > Makefile
-svn add Makefile 
+git cat-file blob 6683463e:Makefile > trunk/Makefile
+svn add trunk/Makefile 
 
-echo "Committing ANCESTOR"
-svn commit -m "ancestor"
-cd ..
+say "Committing ANCESTOR"
+i=$(commit $i "ancestor")
 svn cp trunk branches/left
 
-echo "Committing BRANCH POINT"
-svn commit -m "make left branch"
+say "Committing BRANCH POINT"
+i=$(commit $i "make left branch")
 svn cp trunk branches/right
 
-echo "Committing other BRANCH POINT"
-svn commit -m "make right branch"
-cd branches/left/
+say "Committing other BRANCH POINT"
+i=$(commit $i "make right branch")
 
-#$sm init
-#svn commit -m "init svnmerge"
+say "Committing LEFT UPDATE"
+git cat-file blob 5873b67e:Makefile > branches/left/Makefile
+i=$(commit $i "left update 1")
 
-git cat-file blob 5873b67e:Makefile > Makefile
-echo "Committing BRANCH UPDATE 1"
-svn commit -m "left update 1"
-cd ../..
-
-cd trunk
-git cat-file blob 75118b13:Makefile > Makefile
-echo "Committing TRUNK UPDATE"
-svn commit -m "trunk update"
+git cat-file blob 75118b13:Makefile > branches/right/Makefile
+say "Committing RIGHT UPDATE"
+pre_right_update_1=$i
+i=$(commit $i "right update 1")
 
-cd ../branches/left
-git cat-file blob ff5ebe39:Makefile > Makefile
-echo "Committing BRANCH UPDATE 2"
-svn commit -m "left update 2"
+say "Making more commits on LEFT"
+git cat-file blob ff5ebe39:Makefile > branches/left/Makefile
+i=$(commit $i "left update 2")
+git cat-file blob b5039db6:Makefile > branches/left/Makefile
+i=$(commit $i "left update 3")
 
-git cat-file blob b5039db6:Makefile > Makefile
-echo "Committing BRANCH UPDATE 3"
-svn commit -m "left update 3"
+say "Making a LEFT SUB-BRANCH"
+svn cp branches/left branches/left-sub
+sub_left_make=$i
+i=$(commit $i "make left sub-branch")
 
-# merge to trunk
+say "Making a commit on LEFT SUB-BRANCH"
+echo "crunch" > branches/left-sub/README
+svn add branches/left-sub/README
+i=$(commit $i "left sub-branch update 1")
 
-cd ../..
+say "Merging LEFT to TRUNK"
 svn update
 cd trunk
-
 svn merge ../branches/left --accept postpone
-
-git cat-file blob b51ad431:Makefile > Makefile
-
+git cat-file blob b5039db6:Makefile > Makefile
 svn resolved Makefile
+i=$(commit $i "Merge left to trunk 1")
+cd ..
 
-svn commit -m "Merge trunk 1"
-
-# create commits on both branches
-
-cd ../branches/left
-git cat-file blob ff5ebe39:Makefile > Makefile
-echo "Committing BRANCH UPDATE 4"
-svn commit -m "left update 4"
-
-cd ../right
-git cat-file blob b5039db6:Makefile > Makefile
-echo "Committing other BRANCH UPDATE 1"
-svn commit -m "right update 1"
+say "Making more commits on LEFT and RIGHT"
+echo "touche" > branches/left/zlonk
+svn add branches/left/zlonk
+i=$(commit $i "left update 4")
+echo "thwacke" > branches/right/bang
+svn add branches/right/bang
+i=$(commit $i "right update 2")
 
-# merge to trun again
+say "Squash merge of RIGHT tip 2 commits onto TRUNK"
+svn update
+cd trunk
+svn merge -r$pre_right_update_1:$i ../branches/right
+i=$(commit $i "Cherry-pick right 2 commits to trunk")
+cd ..
 
-cd ../..
+say "Merging RIGHT to TRUNK"
 svn update
 cd trunk
+svn merge ../branches/right --accept postpone
+git cat-file blob b51ad431:Makefile > Makefile
+svn resolved Makefile
+i=$(commit $i "Merge right to trunk 1")
+cd ..
 
-svn merge ../branches/left --accept postpone
+say "Making more commits on RIGHT and TRUNK"
+echo "whamm" > branches/right/urkkk
+svn add branches/right/urkkk
+i=$(commit $i "right update 3")
+echo "pow" > trunk/vronk
+svn add trunk/vronk
+i=$(commit $i "trunk update 1")
 
+say "Merging RIGHT to LEFT SUB-BRANCH"
+svn update
+cd branches/left-sub
+svn merge ../right --accept postpone
 git cat-file blob b51ad431:Makefile > Makefile
-
 svn resolved Makefile
+i=$(commit $i "Merge right to left sub-branch")
+cd ../..
 
-svn commit -m "Merge trunk 2"
+say "Making more commits on LEFT SUB-BRANCH and LEFT"
+echo "zowie" > branches/left-sub/wham_eth
+svn add branches/left-sub/wham_eth
+pre_sub_left_update_2=$i
+i=$(commit $i "left sub-branch update 2")
+sub_left_update_2=$i
+echo "eee_yow" > branches/left/glurpp
+svn add branches/left/glurpp
+i=$(commit $i "left update 5")
+
+say "Cherry pick LEFT SUB-BRANCH commit to LEFT"
+svn update
+cd branches/left
+svn merge -r$pre_sub_left_update_2:$sub_left_update_2 ../left-sub
+i=$(commit $i "Cherry-pick left sub-branch commit to left")
+cd ../..
 
+say "Merging LEFT SUB-BRANCH back to LEFT"
+svn update
+cd branches/left
+# it's only a merge because the previous merge cherry-picked the top commit
+svn merge -r$sub_left_make:$sub_left_update_2 ../left-sub --accept postpone
+i=$(commit $i "Merge left sub-branch to left")
 cd ../..
 
+say "Merging EVERYTHING to TRUNK"
+svn update
+cd trunk
+svn merge ../branches/left --accept postpone
+svn resolved bang
+i=$(commit $i "Merge left to trunk 2")
+# this merge, svn happily updates the mergeinfo, but there is actually
+# nothing to merge.  git-svn will not make a meaningless merge commit.
+svn merge ../branches/right --accept postpone
+i=$(commit $i "non-merge right to trunk 2")
+cd ..
+
+cd ..
 svnadmin dump foo.svn > svn-mergeinfo.dump
 
 rm -rf foo foo.svn
diff --git a/t/t9151/svn-mergeinfo.dump b/t/t9151/svn-mergeinfo.dump
index 11a883f..9543e31 100644
--- a/t/t9151/svn-mergeinfo.dump
+++ b/t/t9151/svn-mergeinfo.dump
@@ -1,6 +1,6 @@
 SVN-fs-dump-format-version: 2
 
-UUID: 1530d5a2-a1dc-4438-8ad5-d95e96db8945
+UUID: 64142547-0943-4db2-836a-d1e1eb2f9924
 
 Revision-number: 0
 Prop-content-length: 56
@@ -9,25 +9,25 @@ Content-length: 56
 K 8
 svn:date
 V 27
-2009-11-12T20:29:38.812226Z
+2009-12-19T16:17:51.232640Z
 PROPS-END
 
 Revision-number: 1
-Prop-content-length: 127
-Content-length: 127
+Prop-content-length: 128
+Content-length: 128
 
 K 7
 svn:log
-V 24
-Setup trunk and branches
+V 29
+(r1) Setup trunk and branches
 K 10
 svn:author
-V 8
-tallsopp
+V 4
+samv
 K 8
 svn:date
 V 27
-2009-11-12T20:29:39.045856Z
+2009-12-19T16:17:51.831965Z
 PROPS-END
 
 Node-path: branches
@@ -49,21 +49,21 @@ PROPS-END
 
 
 Revision-number: 2
-Prop-content-length: 110
-Content-length: 110
+Prop-content-length: 112
+Content-length: 112
 
 K 7
 svn:log
-V 8
-ancestor
+V 13
+(r2) ancestor
 K 10
 svn:author
-V 8
-tallsopp
+V 4
+samv
 K 8
 svn:date
 V 27
-2009-11-12T20:29:40.079587Z
+2009-12-19T16:17:52.300075Z
 PROPS-END
 
 Node-path: trunk/Makefile
@@ -156,21 +156,21 @@ backup: clean
 
 
 Revision-number: 3
-Prop-content-length: 119
-Content-length: 119
+Prop-content-length: 120
+Content-length: 120
 
 K 7
 svn:log
-V 16
-make left branch
+V 21
+(r3) make left branch
 K 10
 svn:author
-V 8
-tallsopp
+V 4
+samv
 K 8
 svn:date
 V 27
-2009-11-12T20:29:42.084439Z
+2009-12-19T16:17:52.768800Z
 PROPS-END
 
 Node-path: branches/left
@@ -190,21 +190,21 @@ Text-copy-source-sha1: 103205ce331f7d64086dba497574734f78439590
 
 
 Revision-number: 4
-Prop-content-length: 120
-Content-length: 120
+Prop-content-length: 121
+Content-length: 121
 
 K 7
 svn:log
-V 17
-make right branch
+V 22
+(r4) make right branch
 K 10
 svn:author
-V 8
-tallsopp
+V 4
+samv
 K 8
 svn:date
 V 27
-2009-11-12T20:29:44.065452Z
+2009-12-19T16:17:53.177879Z
 PROPS-END
 
 Node-path: branches/right
@@ -224,21 +224,21 @@ Text-copy-source-sha1: 103205ce331f7d64086dba497574734f78439590
 
 
 Revision-number: 5
-Prop-content-length: 116
-Content-length: 116
+Prop-content-length: 117
+Content-length: 117
 
 K 7
 svn:log
-V 13
-left update 1
+V 18
+(r5) left update 1
 K 10
 svn:author
-V 8
-tallsopp
+V 4
+samv
 K 8
 svn:date
 V 27
-2009-11-12T20:29:45.066262Z
+2009-12-19T16:17:53.604691Z
 PROPS-END
 
 Node-path: branches/left/Makefile
@@ -329,24 +329,24 @@ backup: clean
 
 
 Revision-number: 6
-Prop-content-length: 115
-Content-length: 115
+Prop-content-length: 118
+Content-length: 118
 
 K 7
 svn:log
-V 12
-trunk update
+V 19
+(r6) right update 1
 K 10
 svn:author
-V 8
-tallsopp
+V 4
+samv
 K 8
 svn:date
 V 27
-2009-11-12T20:29:46.278498Z
+2009-12-19T16:17:54.063555Z
 PROPS-END
 
-Node-path: trunk/Makefile
+Node-path: branches/right/Makefile
 Node-kind: file
 Node-action: change
 Text-content-length: 2521
@@ -437,21 +437,21 @@ backup: clean
 
 
 Revision-number: 7
-Prop-content-length: 116
-Content-length: 116
+Prop-content-length: 117
+Content-length: 117
 
 K 7
 svn:log
-V 13
-left update 2
+V 18
+(r7) left update 2
 K 10
 svn:author
-V 8
-tallsopp
+V 4
+samv
 K 8
 svn:date
 V 27
-2009-11-12T20:29:47.069090Z
+2009-12-19T16:17:54.523904Z
 PROPS-END
 
 Node-path: branches/left/Makefile
@@ -542,21 +542,21 @@ backup: clean
 
 
 Revision-number: 8
-Prop-content-length: 116
-Content-length: 116
+Prop-content-length: 117
+Content-length: 117
 
 K 7
 svn:log
-V 13
-left update 3
+V 18
+(r8) left update 3
 K 10
 svn:author
-V 8
-tallsopp
+V 4
+samv
 K 8
 svn:date
 V 27
-2009-11-12T20:29:48.053835Z
+2009-12-19T16:17:54.975970Z
 PROPS-END
 
 Node-path: branches/left/Makefile
@@ -647,33 +647,285 @@ backup: clean
 
 
 Revision-number: 9
-Prop-content-length: 116
-Content-length: 116
+Prop-content-length: 124
+Content-length: 124
 
 K 7
 svn:log
-V 13
-Merge trunk 1
+V 25
+(r9) make left sub-branch
+K 10
+svn:author
+V 4
+samv
+K 8
+svn:date
+V 27
+2009-12-19T16:17:55.459904Z
+PROPS-END
+
+Node-path: branches/left-sub
+Node-kind: dir
+Node-action: add
+Node-copyfrom-rev: 3
+Node-copyfrom-path: branches/left
+
+
+Node-path: branches/left-sub/Makefile
+Node-kind: file
+Node-action: delete
+
+Node-path: branches/left-sub/Makefile
+Node-kind: file
+Node-action: add
+Node-copyfrom-rev: 8
+Node-copyfrom-path: branches/left/Makefile
+Text-copy-source-md5: 5ccff689fb290e00b85fe18ee50c54ba
+Text-copy-source-sha1: a13de8e23f1483efca3e57b2b64b0ae6f740ce10
+
+
+
+
+Revision-number: 10
+Prop-content-length: 129
+Content-length: 129
+
+K 7
+svn:log
+V 30
+(r10) left sub-branch update 1
 K 10
 svn:author
-V 8
-tallsopp
+V 4
+samv
 K 8
 svn:date
 V 27
-2009-11-12T20:29:51.098306Z
+2009-12-19T16:17:55.862113Z
+PROPS-END
+
+Node-path: branches/left-sub/README
+Node-kind: file
+Node-action: add
+Prop-content-length: 10
+Text-content-length: 7
+Text-content-md5: fdbcfb6be9afe1121862143f226b51cf
+Text-content-sha1: 1d1f5ea4ceb584337ffe59b8980d92e3b78dfef4
+Content-length: 17
+
+PROPS-END
+crunch
+
+
+Revision-number: 11
+Prop-content-length: 126
+Content-length: 126
+
+K 7
+svn:log
+V 27
+(r11) Merge left to trunk 1
+K 10
+svn:author
+V 4
+samv
+K 8
+svn:date
+V 27
+2009-12-19T16:17:56.413416Z
 PROPS-END
 
 Node-path: trunk
 Node-kind: dir
 Node-action: change
-Prop-content-length: 53
-Content-length: 53
+Prop-content-length: 54
+Content-length: 54
 
 K 13
 svn:mergeinfo
-V 18
-/branches/left:2-8
+V 19
+/branches/left:2-10
+PROPS-END
+
+
+Node-path: trunk/Makefile
+Node-kind: file
+Node-action: change
+Text-content-length: 2593
+Text-content-md5: 5ccff689fb290e00b85fe18ee50c54ba
+Text-content-sha1: a13de8e23f1483efca3e57b2b64b0ae6f740ce10
+Content-length: 2593
+
+# -DCOLLISION_CHECK if you believe that SHA1's
+# 1461501637330902918203684832716283019655932542976 hashes do not give you
+# enough guarantees about no collisions between objects ever hapenning.
+#
+# -DNSEC if you want git to care about sub-second file mtimes and ctimes.
+# Note that you need some new glibc (at least >2.2.4) for this, and it will
+# BREAK YOUR LOCAL DIFFS! show-diff and anything using it will likely randomly
+# break unless your underlying filesystem supports those sub-second times
+# (my ext3 doesn't).
+CFLAGS=-g -O3 -Wall
+
+CC=gcc
+
+
+PROG=   update-cache show-diff init-db write-tree read-tree commit-tree \
+	cat-file fsck-cache checkout-cache diff-tree rev-tree show-files \
+	check-files ls-tree merge-base
+
+all: $(PROG)
+
+install: $(PROG)
+	install $(PROG) $(HOME)/bin/
+
+LIBS= -lssl -lz
+
+init-db: init-db.o
+
+update-cache: update-cache.o read-cache.o
+	$(CC) $(CFLAGS) -o update-cache update-cache.o read-cache.o $(LIBS)
+
+show-diff: show-diff.o read-cache.o
+	$(CC) $(CFLAGS) -o show-diff show-diff.o read-cache.o $(LIBS)
+
+write-tree: write-tree.o read-cache.o
+	$(CC) $(CFLAGS) -o write-tree write-tree.o read-cache.o $(LIBS)
+
+read-tree: read-tree.o read-cache.o
+	$(CC) $(CFLAGS) -o read-tree read-tree.o read-cache.o $(LIBS)
+
+commit-tree: commit-tree.o read-cache.o
+	$(CC) $(CFLAGS) -o commit-tree commit-tree.o read-cache.o $(LIBS)
+
+cat-file: cat-file.o read-cache.o
+	$(CC) $(CFLAGS) -o cat-file cat-file.o read-cache.o $(LIBS)
+
+fsck-cache: fsck-cache.o read-cache.o object.o commit.o tree.o blob.o
+	$(CC) $(CFLAGS) -o fsck-cache fsck-cache.o read-cache.o object.o commit.o tree.o blob.o $(LIBS)
+
+checkout-cache: checkout-cache.o read-cache.o
+	$(CC) $(CFLAGS) -o checkout-cache checkout-cache.o read-cache.o $(LIBS)
+
+diff-tree: diff-tree.o read-cache.o
+	$(CC) $(CFLAGS) -o diff-tree diff-tree.o read-cache.o $(LIBS)
+
+rev-tree: rev-tree.o read-cache.o object.o commit.o tree.o blob.o
+	$(CC) $(CFLAGS) -o rev-tree rev-tree.o read-cache.o object.o commit.o tree.o blob.o $(LIBS)
+
+show-files: show-files.o read-cache.o
+	$(CC) $(CFLAGS) -o show-files show-files.o read-cache.o $(LIBS)
+
+check-files: check-files.o read-cache.o
+	$(CC) $(CFLAGS) -o check-files check-files.o read-cache.o $(LIBS)
+
+ls-tree: ls-tree.o read-cache.o
+	$(CC) $(CFLAGS) -o ls-tree ls-tree.o read-cache.o $(LIBS)
+
+merge-base: merge-base.o read-cache.o object.o commit.o tree.o blob.o
+	$(CC) $(CFLAGS) -o merge-base merge-base.o read-cache.o object.o commit.o tree.o blob.o $(LIBS)
+
+read-cache.o: cache.h
+show-diff.o: cache.h
+
+clean:
+	rm -f *.o $(PROG)
+
+backup: clean
+	cd .. ; tar czvf dircache.tar.gz dir-cache
+
+
+Revision-number: 12
+Prop-content-length: 118
+Content-length: 118
+
+K 7
+svn:log
+V 19
+(r12) left update 4
+K 10
+svn:author
+V 4
+samv
+K 8
+svn:date
+V 27
+2009-12-19T16:17:56.831014Z
+PROPS-END
+
+Node-path: branches/left/zlonk
+Node-kind: file
+Node-action: add
+Prop-content-length: 10
+Text-content-length: 7
+Text-content-md5: 8b9d8c7c2aaa6167e7d3407a773bbbba
+Text-content-sha1: 9716527ebd70a75c27625cacbeb2d897c6e86178
+Content-length: 17
+
+PROPS-END
+touche
+
+
+Revision-number: 13
+Prop-content-length: 119
+Content-length: 119
+
+K 7
+svn:log
+V 20
+(r13) right update 2
+K 10
+svn:author
+V 4
+samv
+K 8
+svn:date
+V 27
+2009-12-19T16:17:57.341143Z
+PROPS-END
+
+Node-path: branches/right/bang
+Node-kind: file
+Node-action: add
+Prop-content-length: 10
+Text-content-length: 8
+Text-content-md5: 34c28f1d2dc6a9adeccc4265bf7516cb
+Text-content-sha1: 0bc5bb345c0e71d28f784f12e0bd2d384c283062
+Content-length: 18
+
+PROPS-END
+thwacke
+
+
+Revision-number: 14
+Prop-content-length: 141
+Content-length: 141
+
+K 7
+svn:log
+V 42
+(r14) Cherry-pick right 2 commits to trunk
+K 10
+svn:author
+V 4
+samv
+K 8
+svn:date
+V 27
+2009-12-19T16:17:57.841851Z
+PROPS-END
+
+Node-path: trunk
+Node-kind: dir
+Node-action: change
+Prop-content-length: 75
+Content-length: 75
+
+K 13
+svn:mergeinfo
+V 40
+/branches/left:2-10
+/branches/right:6-13
 PROPS-END
 
 
@@ -767,31 +1019,147 @@ backup: clean
 	cd .. ; tar czvf dircache.tar.gz dir-cache
 
 
-Revision-number: 10
-Prop-content-length: 116
-Content-length: 116
+Node-path: trunk/bang
+Node-kind: file
+Node-action: add
+Node-copyfrom-rev: 13
+Node-copyfrom-path: branches/right/bang
+Text-copy-source-md5: 34c28f1d2dc6a9adeccc4265bf7516cb
+Text-copy-source-sha1: 0bc5bb345c0e71d28f784f12e0bd2d384c283062
+
+
+Revision-number: 15
+Prop-content-length: 127
+Content-length: 127
 
 K 7
 svn:log
-V 13
-left update 4
+V 28
+(r15) Merge right to trunk 1
 K 10
 svn:author
-V 8
-tallsopp
+V 4
+samv
 K 8
 svn:date
 V 27
-2009-11-12T20:29:52.081644Z
+2009-12-19T16:17:58.368520Z
 PROPS-END
 
-Node-path: branches/left/Makefile
+Node-path: trunk
+Node-kind: dir
+Node-action: change
+Prop-content-length: 75
+Content-length: 75
+
+K 13
+svn:mergeinfo
+V 40
+/branches/left:2-10
+/branches/right:2-14
+PROPS-END
+
+
+Revision-number: 16
+Prop-content-length: 119
+Content-length: 119
+
+K 7
+svn:log
+V 20
+(r16) right update 3
+K 10
+svn:author
+V 4
+samv
+K 8
+svn:date
+V 27
+2009-12-19T16:17:58.779056Z
+PROPS-END
+
+Node-path: branches/right/urkkk
+Node-kind: file
+Node-action: add
+Prop-content-length: 10
+Text-content-length: 6
+Text-content-md5: 5889c8392e16251b0c80927607a03036
+Text-content-sha1: 3934264d277a0cf886b6b1c7f2b9e56da2525302
+Content-length: 16
+
+PROPS-END
+whamm
+
+
+Revision-number: 17
+Prop-content-length: 119
+Content-length: 119
+
+K 7
+svn:log
+V 20
+(r17) trunk update 1
+K 10
+svn:author
+V 4
+samv
+K 8
+svn:date
+V 27
+2009-12-19T16:17:59.221851Z
+PROPS-END
+
+Node-path: trunk/vronk
 Node-kind: file
+Node-action: add
+Prop-content-length: 10
+Text-content-length: 4
+Text-content-md5: b2f80fa02a7f1364b9c29d3da44bf9f9
+Text-content-sha1: e994d980c0f2d7a3f76138bf96d57f36f9633828
+Content-length: 14
+
+PROPS-END
+pow
+
+
+Revision-number: 18
+Prop-content-length: 135
+Content-length: 135
+
+K 7
+svn:log
+V 36
+(r18) Merge right to left sub-branch
+K 10
+svn:author
+V 4
+samv
+K 8
+svn:date
+V 27
+2009-12-19T16:17:59.781666Z
+PROPS-END
+
+Node-path: branches/left-sub
+Node-kind: dir
 Node-action: change
-Text-content-length: 2529
-Text-content-md5: f6b197cc3f2e89a83e545d4bb003de73
-Text-content-sha1: 2f656677cfec0bceec85e53036ffb63e25126f8e
-Content-length: 2529
+Prop-content-length: 55
+Content-length: 55
+
+K 13
+svn:mergeinfo
+V 20
+/branches/right:2-17
+PROPS-END
+
+
+Node-path: branches/left-sub/Makefile
+Node-kind: file
+Node-action: change
+Text-content-length: 2713
+Text-content-md5: 0afbe34f244cd662b1f97d708c687f90
+Text-content-sha1: 46d9377d783e67a9b581da110352e799517c8a14
+Content-length: 2713
 
 # -DCOLLISION_CHECK if you believe that SHA1's
 # 1461501637330902918203684832716283019655932542976 hashes do not give you
@@ -809,7 +1177,7 @@ CC=gcc
 
 PROG=   update-cache show-diff init-db write-tree read-tree commit-tree \
 	cat-file fsck-cache checkout-cache diff-tree rev-tree show-files \
-	check-files ls-tree merge-base
+	check-files ls-tree merge-base merge-cache
 
 all: $(PROG)
 
@@ -859,8 +1227,11 @@ check-files: check-files.o read-cache.o
 ls-tree: ls-tree.o read-cache.o
 	$(CC) $(CFLAGS) -o ls-tree ls-tree.o read-cache.o $(LIBS)
 
-merge-base: merge-base.o read-cache.o
-	$(CC) $(CFLAGS) -o merge-base merge-base.o read-cache.o $(LIBS)
+merge-base: merge-base.o read-cache.o object.o commit.o tree.o blob.o
+	$(CC) $(CFLAGS) -o merge-base merge-base.o read-cache.o object.o commit.o tree.o blob.o $(LIBS)
+
+merge-cache: merge-cache.o read-cache.o
+	$(CC) $(CFLAGS) -o merge-cache merge-cache.o read-cache.o $(LIBS)
 
 read-cache.o: cache.h
 show-diff.o: cache.h
@@ -872,31 +1243,165 @@ backup: clean
 	cd .. ; tar czvf dircache.tar.gz dir-cache
 
 
-Revision-number: 11
-Prop-content-length: 117
-Content-length: 117
+Node-path: branches/left-sub/bang
+Node-kind: file
+Node-action: add
+Node-copyfrom-rev: 17
+Node-copyfrom-path: branches/right/bang
+Text-copy-source-md5: 34c28f1d2dc6a9adeccc4265bf7516cb
+Text-copy-source-sha1: 0bc5bb345c0e71d28f784f12e0bd2d384c283062
+
+
+Node-path: branches/left-sub/urkkk
+Node-kind: file
+Node-action: add
+Node-copyfrom-rev: 17
+Node-copyfrom-path: branches/right/urkkk
+Text-copy-source-md5: 5889c8392e16251b0c80927607a03036
+Text-copy-source-sha1: 3934264d277a0cf886b6b1c7f2b9e56da2525302
+
+
+Revision-number: 19
+Prop-content-length: 129
+Content-length: 129
 
 K 7
 svn:log
-V 14
-right update 1
+V 30
+(r19) left sub-branch update 2
 K 10
 svn:author
-V 8
-tallsopp
+V 4
+samv
 K 8
 svn:date
 V 27
-2009-11-12T20:29:53.059636Z
+2009-12-19T16:18:00.200531Z
 PROPS-END
 
-Node-path: branches/right/Makefile
+Node-path: branches/left-sub/wham_eth
+Node-kind: file
+Node-action: add
+Prop-content-length: 10
+Text-content-length: 6
+Text-content-md5: 757bcd5818572ef3f9580052617c1c8b
+Text-content-sha1: b165019b005c199237ba822c4404e771e93b654a
+Content-length: 16
+
+PROPS-END
+zowie
+
+
+Revision-number: 20
+Prop-content-length: 118
+Content-length: 118
+
+K 7
+svn:log
+V 19
+(r20) left update 5
+K 10
+svn:author
+V 4
+samv
+K 8
+svn:date
+V 27
+2009-12-19T16:18:00.659636Z
+PROPS-END
+
+Node-path: branches/left/glurpp
 Node-kind: file
+Node-action: add
+Prop-content-length: 10
+Text-content-length: 8
+Text-content-md5: 14a169f628e0bb59df9c2160649d0a30
+Text-content-sha1: ef7d929e52177767ecfcd28941f6b7f04b4131e3
+Content-length: 18
+
+PROPS-END
+eee_yow
+
+
+Revision-number: 21
+Prop-content-length: 147
+Content-length: 147
+
+K 7
+svn:log
+V 48
+(r21) Cherry-pick left sub-branch commit to left
+K 10
+svn:author
+V 4
+samv
+K 8
+svn:date
+V 27
+2009-12-19T16:18:01.194402Z
+PROPS-END
+
+Node-path: branches/left
+Node-kind: dir
 Node-action: change
-Text-content-length: 2593
-Text-content-md5: 5ccff689fb290e00b85fe18ee50c54ba
-Text-content-sha1: a13de8e23f1483efca3e57b2b64b0ae6f740ce10
-Content-length: 2593
+Prop-content-length: 56
+Content-length: 56
+
+K 13
+svn:mergeinfo
+V 21
+/branches/left-sub:19
+PROPS-END
+
+
+Node-path: branches/left/wham_eth
+Node-kind: file
+Node-action: add
+Node-copyfrom-rev: 19
+Node-copyfrom-path: branches/left-sub/wham_eth
+Text-copy-source-md5: 757bcd5818572ef3f9580052617c1c8b
+Text-copy-source-sha1: b165019b005c199237ba822c4404e771e93b654a
+
+
+Revision-number: 22
+Prop-content-length: 134
+Content-length: 134
+
+K 7
+svn:log
+V 35
+(r22) Merge left sub-branch to left
+K 10
+svn:author
+V 4
+samv
+K 8
+svn:date
+V 27
+2009-12-19T16:18:01.679218Z
+PROPS-END
+
+Node-path: branches/left
+Node-kind: dir
+Node-action: change
+Prop-content-length: 79
+Content-length: 79
+
+K 13
+svn:mergeinfo
+V 44
+/branches/left-sub:4-19
+/branches/right:2-17
+PROPS-END
+
+
+Node-path: branches/left/Makefile
+Node-kind: file
+Node-action: change
+Text-content-length: 2713
+Text-content-md5: 0afbe34f244cd662b1f97d708c687f90
+Text-content-sha1: 46d9377d783e67a9b581da110352e799517c8a14
+Content-length: 2713
 
 # -DCOLLISION_CHECK if you believe that SHA1's
 # 1461501637330902918203684832716283019655932542976 hashes do not give you
@@ -914,7 +1419,7 @@ CC=gcc
 
 PROG=   update-cache show-diff init-db write-tree read-tree commit-tree \
 	cat-file fsck-cache checkout-cache diff-tree rev-tree show-files \
-	check-files ls-tree merge-base
+	check-files ls-tree merge-base merge-cache
 
 all: $(PROG)
 
@@ -967,6 +1472,9 @@ ls-tree: ls-tree.o read-cache.o
 merge-base: merge-base.o read-cache.o object.o commit.o tree.o blob.o
 	$(CC) $(CFLAGS) -o merge-base merge-base.o read-cache.o object.o commit.o tree.o blob.o $(LIBS)
 
+merge-cache: merge-cache.o read-cache.o
+	$(CC) $(CFLAGS) -o merge-cache merge-cache.o read-cache.o $(LIBS)
+
 read-cache.o: cache.h
 show-diff.o: cache.h
 
@@ -977,34 +1485,141 @@ backup: clean
 	cd .. ; tar czvf dircache.tar.gz dir-cache
 
 
-Revision-number: 12
-Prop-content-length: 116
-Content-length: 116
+Node-path: branches/left/README
+Node-kind: file
+Node-action: add
+Node-copyfrom-rev: 18
+Node-copyfrom-path: branches/left-sub/README
+Text-copy-source-md5: fdbcfb6be9afe1121862143f226b51cf
+Text-copy-source-sha1: 1d1f5ea4ceb584337ffe59b8980d92e3b78dfef4
+
+
+Node-path: branches/left/bang
+Node-kind: file
+Node-action: add
+Node-copyfrom-rev: 18
+Node-copyfrom-path: branches/left-sub/bang
+Text-copy-source-md5: 34c28f1d2dc6a9adeccc4265bf7516cb
+Text-copy-source-sha1: 0bc5bb345c0e71d28f784f12e0bd2d384c283062
+
+
+Node-path: branches/left/urkkk
+Node-kind: file
+Node-action: add
+Node-copyfrom-rev: 18
+Node-copyfrom-path: branches/left-sub/urkkk
+Text-copy-source-md5: 5889c8392e16251b0c80927607a03036
+Text-copy-source-sha1: 3934264d277a0cf886b6b1c7f2b9e56da2525302
+
+
+Revision-number: 23
+Prop-content-length: 126
+Content-length: 126
 
 K 7
 svn:log
-V 13
-Merge trunk 2
+V 27
+(r23) Merge left to trunk 2
 K 10
 svn:author
-V 8
-tallsopp
+V 4
+samv
 K 8
 svn:date
 V 27
-2009-11-12T20:29:56.083003Z
+2009-12-19T16:18:02.212349Z
 PROPS-END
 
 Node-path: trunk
 Node-kind: dir
 Node-action: change
-Prop-content-length: 54
-Content-length: 54
+Prop-content-length: 99
+Content-length: 99
 
 K 13
 svn:mergeinfo
-V 19
-/branches/left:2-11
+V 64
+/branches/left:2-22
+/branches/left-sub:4-19
+/branches/right:2-17
+PROPS-END
+
+
+Node-path: trunk/README
+Node-kind: file
+Node-action: add
+Node-copyfrom-rev: 22
+Node-copyfrom-path: branches/left/README
+Text-copy-source-md5: fdbcfb6be9afe1121862143f226b51cf
+Text-copy-source-sha1: 1d1f5ea4ceb584337ffe59b8980d92e3b78dfef4
+
+
+Node-path: trunk/glurpp
+Node-kind: file
+Node-action: add
+Node-copyfrom-rev: 22
+Node-copyfrom-path: branches/left/glurpp
+Text-copy-source-md5: 14a169f628e0bb59df9c2160649d0a30
+Text-copy-source-sha1: ef7d929e52177767ecfcd28941f6b7f04b4131e3
+
+
+Node-path: trunk/urkkk
+Node-kind: file
+Node-action: add
+Node-copyfrom-rev: 22
+Node-copyfrom-path: branches/left/urkkk
+Text-copy-source-md5: 5889c8392e16251b0c80927607a03036
+Text-copy-source-sha1: 3934264d277a0cf886b6b1c7f2b9e56da2525302
+
+
+Node-path: trunk/wham_eth
+Node-kind: file
+Node-action: add
+Node-copyfrom-rev: 22
+Node-copyfrom-path: branches/left/wham_eth
+Text-copy-source-md5: 757bcd5818572ef3f9580052617c1c8b
+Text-copy-source-sha1: b165019b005c199237ba822c4404e771e93b654a
+
+
+Node-path: trunk/zlonk
+Node-kind: file
+Node-action: add
+Node-copyfrom-rev: 22
+Node-copyfrom-path: branches/left/zlonk
+Text-copy-source-md5: 8b9d8c7c2aaa6167e7d3407a773bbbba
+Text-copy-source-sha1: 9716527ebd70a75c27625cacbeb2d897c6e86178
+
+
+Revision-number: 24
+Prop-content-length: 131
+Content-length: 131
+
+K 7
+svn:log
+V 32
+(r24) non-merge right to trunk 2
+K 10
+svn:author
+V 4
+samv
+K 8
+svn:date
+V 27
+2009-12-19T16:18:02.672148Z
+PROPS-END
+
+Node-path: trunk
+Node-kind: dir
+Node-action: change
+Prop-content-length: 99
+Content-length: 99
+
+K 13
+svn:mergeinfo
+V 64
+/branches/left:2-22
+/branches/left-sub:4-19
+/branches/right:2-22
 PROPS-END
 
 
-- 
1.6.3.3

[PATCH 2/5] git-svn: memoize conversion of SVN merge ticket info to git commit ranges

[Sam Vilain]

Each time the svn mergeinfo ticket changes, we look it up in the rev_map;
when there are a lot of merged branches, this will result in many repeated
lookups of the same information for subsequent commits.  Arrange the slow
part of the function so that it may be memoized, and memoize it.  The more
expensive revision walking operation can be memoized separately.

Signed-off-by: Sam Vilain <sam@vilain.net>
---
 git-svn.perl |   91 ++++++++++++++++++++++++++++++++++-----------------------
 1 files changed, 54 insertions(+), 37 deletions(-)

diff --git a/git-svn.perl b/git-svn.perl
index a4b052c..3b17a83 100755
--- a/git-svn.perl
+++ b/git-svn.perl
@@ -1634,6 +1634,7 @@ use Carp qw/croak/;
 use File::Path qw/mkpath/;
 use File::Copy qw/copy/;
 use IPC::Open3;
+use Memoize;  # core since 5.8.0, Jul 2002
 
 my ($_gc_nr, $_gc_period);
 
@@ -2967,6 +2968,55 @@ sub find_extra_svk_parents {
 	}
 }
 
+sub lookup_svn_merge {
+	my $uuid = shift;
+	my $url = shift;
+	my $merge = shift;
+
+	my ($source, $revs) = split ":", $merge;
+	my $path = $source;
+	$path =~ s{^/}{};
+	my $gs = Git::SVN->find_by_url($url.$source, $url, $path);
+	if ( !$gs ) {
+		warn "Couldn't find revmap for $url$source\n";
+		next;
+	}
+	my @ranges = split ",", $revs;
+	my ($tip, $tip_commit);
+	my @merged_commit_ranges;
+	# find the tip
+	for my $range ( @ranges ) {
+		my ($bottom, $top) = split "-", $range;
+		$top ||= $bottom;
+		my $bottom_commit =
+			$gs->rev_map_get($bottom, $uuid) ||
+			$gs->rev_map_get($bottom+1, $uuid);
+		my $top_commit;
+		for (; !$top_commit && $top >= $bottom; --$top) {
+			$top_commit =
+				$gs->rev_map_get($top, $uuid);
+		}
+
+		unless ($top_commit and $bottom_commit) {
+			warn "W:unknown path/rev in svn:mergeinfo "
+				."dirprop: $source:$range\n";
+			next;
+		}
+
+		push @merged_commit_ranges,
+			"$bottom_commit..$top_commit";
+
+		if ( !defined $tip or $top > $tip ) {
+			$tip = $top;
+			$tip_commit = $top_commit;
+		}
+	}
+	return ($tip_commit, @merged_commit_ranges);
+}
+BEGIN {
+	memoize 'lookup_svn_merge';
+}
+
 # note: this function should only be called if the various dirprops
 # have actually changed
 sub find_extra_svn_parents {
@@ -2981,44 +3031,11 @@ sub find_extra_svn_parents {
 	my @merge_tips;
 	my @merged_commit_ranges;
 	my $url = $self->rewrite_root || $self->{url};
+	my $uuid = $self->ra_uuid;
 	for my $merge ( @merges ) {
-		my ($source, $revs) = split ":", $merge;
-		my $path = $source;
-		$path =~ s{^/}{};
-		my $gs = Git::SVN->find_by_url($url.$source, $url, $path);
-		if ( !$gs ) {
-			warn "Couldn't find revmap for $url$source\n";
-			next;
-		}
-		my @ranges = split ",", $revs;
-		my ($tip, $tip_commit);
-		# find the tip
-		for my $range ( @ranges ) {
-			my ($bottom, $top) = split "-", $range;
-			$top ||= $bottom;
-			my $bottom_commit =
-				$gs->rev_map_get($bottom, $self->ra_uuid) ||
-				$gs->rev_map_get($bottom+1, $self->ra_uuid);
-			my $top_commit;
-			for (; !$top_commit && $top >= $bottom; --$top) {
-				$top_commit =
-					$gs->rev_map_get($top, $self->ra_uuid);
-			}
-
-			unless ($top_commit and $bottom_commit) {
-				warn "W:unknown path/rev in svn:mergeinfo "
-					."dirprop: $source:$range\n";
-				next;
-			}
-
-			push @merged_commit_ranges,
-				"$bottom_commit..$top_commit";
-
-			if ( !defined $tip or $top > $tip ) {
-				$tip = $top;
-				$tip_commit = $top_commit;
-			}
-		}
+		my ($tip_commit, @ranges) =
+			lookup_svn_merge( $uuid, $url, $merge );
+		push @merged_commit_ranges, @ranges;
 		unless (!$tip_commit or
 				grep { $_ eq $tip_commit } @$parents ) {
 			push @merge_tips, $tip_commit;
-- 
1.6.3.3

[PATCH 3/5] git-svn: fix some mistakes with interpreting SVN mergeinfo commit ranges

[Sam Vilain]

SVN's list of commit ranges in mergeinfo tickets is inclusive, whereas
git commit ranges are exclusive on the left hand side.  Also, the end
points of the commit ranges may not exist; they simply delineate
ranges of commits which may or may not exist.  Fix these two mistakes.

Signed-off-by: Sam Vilain <sam@vilain.net>
---
 git-svn.perl             |   12 +++---------
 t/t9151-svn-mergeinfo.sh |    2 +-
 2 files changed, 4 insertions(+), 10 deletions(-)

diff --git a/git-svn.perl b/git-svn.perl
index 3b17a83..9cf4a3e 100755
--- a/git-svn.perl
+++ b/git-svn.perl
@@ -2988,14 +2988,8 @@ sub lookup_svn_merge {
 	for my $range ( @ranges ) {
 		my ($bottom, $top) = split "-", $range;
 		$top ||= $bottom;
-		my $bottom_commit =
-			$gs->rev_map_get($bottom, $uuid) ||
-			$gs->rev_map_get($bottom+1, $uuid);
-		my $top_commit;
-		for (; !$top_commit && $top >= $bottom; --$top) {
-			$top_commit =
-				$gs->rev_map_get($top, $uuid);
-		}
+		my $bottom_commit = $gs->find_rev_after( $bottom, 1, $top );
+		my $top_commit = $gs->find_rev_before( $top, 1, $bottom );
 
 		unless ($top_commit and $bottom_commit) {
 			warn "W:unknown path/rev in svn:mergeinfo "
@@ -3004,7 +2998,7 @@ sub lookup_svn_merge {
 		}
 
 		push @merged_commit_ranges,
-			"$bottom_commit..$top_commit";
+			"$bottom_commit^..$top_commit";
 
 		if ( !defined $tip or $top > $tip ) {
 			$tip = $top;
diff --git a/t/t9151-svn-mergeinfo.sh b/t/t9151-svn-mergeinfo.sh
index dc3478f..f6e00ea 100755
--- a/t/t9151-svn-mergeinfo.sh
+++ b/t/t9151-svn-mergeinfo.sh
@@ -33,7 +33,7 @@ test_expect_success 'svn non-merge merge commits did not become git merge commit
 	[ -z "$bad_non_merges" ]
 	'
 
-test_expect_failure 'everything got merged in the end' '
+test_expect_success 'everything got merged in the end' '
 	unmerged=$(git rev-list --all --not master)
 	[ -z "$unmerged" ]
 	'
-- 
1.6.3.3

[PATCH 4/5] git-svn: exclude already merged tips using one rev-list call

[Sam Vilain]

The old function would have to check all mentioned merge tips, every time
that the mergeinfo ticket changed.  This involved 1-2 rev-list operation
for each listed mergeinfo line.  If there are a lot of feature branches
being merged into a trunk, this makes for a very expensive operation for
detecting the new parents on every merge.

This new version first uses a single 'rev-list' to figure out which commit
ranges are already reachable from the parents.  This is used to eliminate
the already merged branches from the list.

Signed-off-by: Sam Vilain <sam@vilain.net>
---
 git-svn.perl |   52 ++++++++++++++++++++++++++++++++++++++++++++++++----
 1 files changed, 48 insertions(+), 4 deletions(-)

diff --git a/git-svn.perl b/git-svn.perl
index 9cf4a3e..7a790d7 100755
--- a/git-svn.perl
+++ b/git-svn.perl
@@ -3011,6 +3011,41 @@ BEGIN {
 	memoize 'lookup_svn_merge';
 }
 
+sub parents_exclude {
+	my $parents = shift;
+	my @commits = @_;
+	return unless @commits;
+
+	my @excluded;
+	my $excluded;
+	do {
+		my @cmd = ('rev-list', "-1", @commits, "--not", @$parents );
+		$excluded = command_oneline(@cmd);
+		if ( $excluded ) {
+			my @new;
+			my $found;
+			for my $commit ( @commits ) {
+				if ( $commit eq $excluded ) {
+					push @excluded, $commit;
+					$found++;
+					last;
+				}
+				else {
+					push @new, $commit;
+				}
+			}
+			die "saw commit '$excluded' in rev-list output, "
+				."but we didn't ask for that commit (wanted: @commits --not @$parents)"
+					unless $found;
+			@commits = @new;
+		}
+	}
+		while ($excluded and @commits);
+
+	return @excluded;
+}
+
+
 # note: this function should only be called if the various dirprops
 # have actually changed
 sub find_extra_svn_parents {
@@ -3023,23 +3058,32 @@ sub find_extra_svn_parents {
 	# are now marked as merge, we can add the tip as a parent.
 	my @merges = split "\n", $mergeinfo;
 	my @merge_tips;
-	my @merged_commit_ranges;
 	my $url = $self->rewrite_root || $self->{url};
 	my $uuid = $self->ra_uuid;
+	my %ranges;
 	for my $merge ( @merges ) {
 		my ($tip_commit, @ranges) =
 			lookup_svn_merge( $uuid, $url, $merge );
-		push @merged_commit_ranges, @ranges;
 		unless (!$tip_commit or
 				grep { $_ eq $tip_commit } @$parents ) {
 			push @merge_tips, $tip_commit;
+			$ranges{$tip_commit} = \@ranges;
 		} else {
 			push @merge_tips, undef;
 		}
 	}
+
+	my %excluded = map { $_ => 1 }
+		parents_exclude($parents, grep { defined } @merge_tips);
+
+	# check merge tips for new parents
+	my @new_parents;
 	for my $merge_tip ( @merge_tips ) {
 		my $spec = shift @merges;
-		next unless $merge_tip;
+		next unless $merge_tip and $excluded{$merge_tip};
+
+		my $ranges = $ranges{$merge_tip};
+
 		my @cmd = ('rev-list', "-1", $merge_tip,
 			   "--not", @$parents );
 		my ($msg_fh, $ctx) = command_output_pipe(@cmd);
@@ -3049,7 +3093,7 @@ sub find_extra_svn_parents {
 		}
 		command_close_pipe($msg_fh, $ctx);
 		if ( $new ) {
-			push @cmd, @merged_commit_ranges;
+			push @cmd, @$ranges;
 			my ($msg_fh, $ctx) = command_output_pipe(@cmd);
 			my $unmerged;
 			while ( <$msg_fh> ) {
-- 
1.6.3.3

[PATCH 5/5] git-svn: detect cherry-picks correctly.

[Sam Vilain]

The old function was incorrect; in some instances it marks a cherry picked
range as a merged branch (because of an incorrect assumption that
'rev-list COMMIT --not RANGE' would work).  This is replaced with a
function which should detect them correctly, memoized to limit the expense
of dealing with branches with many cherry picks to one 'merge-base' call
per merge, per branch which used cherry picking.

Signed-off-by: Sam Vilain <sam@vilain.net>
---
 git-svn.perl             |   87 +++++++++++++++++++++++++++++++++------------
 t/t9151-svn-mergeinfo.sh |    4 +-
 2 files changed, 66 insertions(+), 25 deletions(-)

diff --git a/git-svn.perl b/git-svn.perl
index 7a790d7..f06e535 100755
--- a/git-svn.perl
+++ b/git-svn.perl
@@ -3007,8 +3007,35 @@ sub lookup_svn_merge {
 	}
 	return ($tip_commit, @merged_commit_ranges);
 }
+
+sub _rev_list {
+	my ($msg_fh, $ctx) = command_output_pipe(
+		"rev-list", @_,
+	       );
+	my @rv;
+	while ( <$msg_fh> ) {
+		chomp;
+		push @rv, $_;
+	}
+	command_close_pipe($msg_fh, $ctx);
+	@rv;
+}
+
+sub check_cherry_pick {
+	my $base = shift;
+	my $tip = shift;
+	my @ranges = @_;
+	my %commits = map { $_ => 1 }
+		_rev_list("--no-merges", $tip, "--not", $base);
+	for my $range ( @ranges ) {
+		delete @commits{_rev_list($range)};
+	}
+	return (keys %commits);
+}
+
 BEGIN {
 	memoize 'lookup_svn_merge';
+	memoize 'check_cherry_pick';
 }
 
 sub parents_exclude {
@@ -3084,32 +3111,46 @@ sub find_extra_svn_parents {
 
 		my $ranges = $ranges{$merge_tip};
 
-		my @cmd = ('rev-list', "-1", $merge_tip,
-			   "--not", @$parents );
-		my ($msg_fh, $ctx) = command_output_pipe(@cmd);
-		my $new;
-		while ( <$msg_fh> ) {
-			$new=1;last;
-		}
-		command_close_pipe($msg_fh, $ctx);
-		if ( $new ) {
-			push @cmd, @$ranges;
-			my ($msg_fh, $ctx) = command_output_pipe(@cmd);
-			my $unmerged;
-			while ( <$msg_fh> ) {
-				$unmerged=1;last;
-			}
-			command_close_pipe($msg_fh, $ctx);
-			if ( $unmerged ) {
-				warn "W:svn cherry-pick ignored ($spec)\n";
-			} else {
-				warn
-				  "Found merge parent (svn:mergeinfo prop): ",
-				  $merge_tip, "\n";
-				push @$parents, $merge_tip;
+		# check out 'new' tips
+		my $merge_base = command_oneline(
+			"merge-base",
+			@$parents, $merge_tip,
+		       );
+
+		# double check that there are no missing non-merge commits
+		my (@incomplete) = check_cherry_pick(
+			$merge_base, $merge_tip,
+			@$ranges,
+		       );
+
+		if ( @incomplete ) {
+			warn "W:svn cherry-pick ignored ($spec) - missing "
+				.@incomplete." commit(s) (eg $incomplete[0])\n";
+		} else {
+			warn
+				"Found merge parent (svn:mergeinfo prop): ",
+					$merge_tip, "\n";
+			push @new_parents, $merge_tip;
+		}
+	}
+
+	# cater for merges which merge commits from multiple branches
+	if ( @new_parents > 1 ) {
+		for ( my $i = 0; $i <= $#new_parents; $i++ ) {
+			for ( my $j = 0; $j <= $#new_parents; $j++ ) {
+				next if $i == $j;
+				next unless $new_parents[$i];
+				next unless $new_parents[$j];
+				my $revs = command_oneline(
+					"rev-list", "-1", "$i..$j",
+				       );
+				if ( !$revs ) {
+					undef($new_parents[$i]);
+				}
 			}
 		}
 	}
+	push @$parents, grep { defined } @new_parents;
 }
 
 sub make_log_entry {
diff --git a/t/t9151-svn-mergeinfo.sh b/t/t9151-svn-mergeinfo.sh
index f6e00ea..359eeaa 100755
--- a/t/t9151-svn-mergeinfo.sh
+++ b/t/t9151-svn-mergeinfo.sh
@@ -15,13 +15,13 @@ test_expect_success 'load svn dump' "
 	git svn fetch --all
 	"
 
-test_expect_failure 'all svn merges became git merge commits' '
+test_expect_success 'all svn merges became git merge commits' '
 	unmarked=$(git rev-list --parents --all --grep=Merge |
 		grep -v " .* " | cut -f1 -d" ")
 	[ -z "$unmarked" ]
 	'
 
-test_expect_failure 'cherry picks did not become git merge commits' '
+test_expect_success 'cherry picks did not become git merge commits' '
 	bad_cherries=$(git rev-list --parents --all --grep=Cherry |
 		grep " .* " | cut -f1 -d" ")
 	[ -z "$bad_cherries" ]
-- 
1.6.3.3

Re: [PATCH 2/5] git-svn: memoize conversion of SVN merge ticket info to git commit ranges

[Sam Vilain]

On Sun, 2009-12-20 at 05:33 +1300, Sam Vilain wrote:
> Each time the svn mergeinfo ticket changes, we look it up in the rev_map;
> when there are a lot of merged branches, this will result in many repeated
> lookups of the same information for subsequent commits.  Arrange the slow
> part of the function so that it may be memoized, and memoize it.  The more
> expensive revision walking operation can be memoized separately.

Sorry, that text was an old revision.  Read as:

Each time the svn mergeinfo ticket changes, we look it up in the
rev_map; when there are a lot of merged branches, this will result in
many repeated lookups of the same information for subsequent commits.
Arrange that part of the function so that it may be memoized, and
memoize it.

Sam

Re: Efficiency and correctness patches for git-svn mergeinfo support

[Sam Vilain]

On Sun, 2009-12-20 at 05:33 +1300, Sam Vilain wrote:
> This series implements some efficiency enhancements; particularly in
> dealing with repositories which have a single trunk branch which
> receives many merges of feature branches and/or cherry-picks.  It also
> fixes a number of corner cases in the merge conversion code.
> 
> It would be nice if the people who have experienced slow git-svn
> performance in these situations could test that this fixes the
> performance issues, and that the resulting repositories seem to have
> correct contents.
> 
> Eric, I'm using Alex's trick of a single commit which adds failing
> tests - marked as _expect_failure - and then marked them as succeeding
> as the series progresses.

More useful info:

 git-svn.perl               |  220 +++++++++----
 t/t9151-svn-mergeinfo.sh   |   27 ++-
 t/t9151/make-svnmerge-dump |  166 ++++++---
 t/t9151/svn-mergeinfo.dump |  839
++++++++++++++++++++++++++++++++++++++------
 4 files changed, 1018 insertions(+), 234 deletions(-)

And available at git://github.com/samv/git (branch mergeinfo-fixes)

Sam

Re: Efficiency and correctness patches for git-svn mergeinfo support

[Andrew Myrick]

> It would be nice if the people who have experienced slow git-svn
> performance in these situations could test that this fixes the
> performance issues, and that the resulting repositories seem to have
> correct contents.

I tried cloning from a fairly recent revision that I knew was after
our switchover to svn 1.5, and I received a number of these errors:

   Couldn't find revmap for [branch]
   Exiting subroutine via next at /Users/adm/libexec/git-core/git-svn line 2983.
   Exiting subroutine via next at /Users/adm/libexec/git-core/git-svn line 2983.
   Exiting subroutine via next at /Users/adm/libexec/git-core/git-svn line 2983.

I'm not sure if this is expected, since I didn't clone from the whole
repo, but it did cause a lot of spew.  I'm starting a fresh clone now,
but it takes a few days to get through the whole repository.  I'm
fairly new to git, so I would welcome any tips on how I can test this
more quickly.

-Andrew

Re: Efficiency and correctness patches for git-svn mergeinfo support

[Sam Vilain]

On Sat, 2009-12-19 at 14:15 -0800, Andrew Myrick wrote:
> I tried cloning from a fairly recent revision that I knew was after
> our switchover to svn 1.5, and I received a number of these errors:
> 
>    Couldn't find revmap for [branch]
>    Exiting subroutine via next at /Users/adm/libexec/git-core/git-svn line 2983.
>    Exiting subroutine via next at /Users/adm/libexec/git-core/git-svn line 2983.
>    Exiting subroutine via next at /Users/adm/libexec/git-core/git-svn line 2983.
> 
> I'm not sure if this is expected, since I didn't clone from the whole
> repo, but it did cause a lot of spew.  I'm starting a fresh clone now,
> but it takes a few days to get through the whole repository.  I'm
> fairly new to git, so I would welcome any tips on how I can test this
> more quickly.

Whoops, no, not expected, I'll post a minor correction.  That means that
the branch which was merged in does not have git-svn metadata; ie, it's
not being tracked explicitly.  If people are doing merging of things
which aren't roots of branches you would expect this.  SVN, like
Perforce, supports a confusing amount of flexibility in its merge
tracking.  If [branch] is a real branch, then you'll want to see why it
doesn't have metadata yet.  Is it really a sub-tree of a real branch?
You could fetch it independently using a separate git-svn remote, or you
could ignore the warning; it should be relatively self-evident what
happened from the merge message and the contents of the changeset.

Note if your repository was significantly re-organized at any point, it
will pay to treat each section of history as a separate import project,
and stitch the results together afterwards using grafts and
filter-branch.

This version should be *significantly* faster than the old one.  ie, it
should not take a minute per commit while importing the heavily
merged-into integration branch.  Possibly a few seconds at most.

Sam

Re: [PATCH 2/5] git-svn: memoize conversion of SVN merge ticket info to git commit ranges

[Sam Vilain]

On Sun, 2009-12-20 at 05:33 +1300, Sam Vilain wrote:
> Each time the svn mergeinfo ticket changes, we look it up in the rev_map;
> when there are a lot of merged branches, this will result in many repeated
> lookups of the same information for subsequent commits.  Arrange the slow
> part of the function so that it may be memoized, and memoize it.  The more
> expensive revision walking operation can be memoized separately.
> 
> Signed-off-by: Sam Vilain <sam@vilain.net>
> ---
>  git-svn.perl |   91 ++++++++++++++++++++++++++++++++++-----------------------
>  1 files changed, 54 insertions(+), 37 deletions(-)
> 
> diff --git a/git-svn.perl b/git-svn.perl
> index a4b052c..3b17a83 100755
> --- a/git-svn.perl
> +++ b/git-svn.perl
> @@ -1634,6 +1634,7 @@ use Carp qw/croak/;
>  use File::Path qw/mkpath/;
>  use File::Copy qw/copy/;
>  use IPC::Open3;
> +use Memoize;  # core since 5.8.0, Jul 2002
>  
>  my ($_gc_nr, $_gc_period);
>  
> @@ -2967,6 +2968,55 @@ sub find_extra_svk_parents {
>  	}
>  }
>  
> +sub lookup_svn_merge {
> +	my $uuid = shift;
> +	my $url = shift;
> +	my $merge = shift;
> +
> +	my ($source, $revs) = split ":", $merge;
> +	my $path = $source;
> +	$path =~ s{^/}{};
> +	my $gs = Git::SVN->find_by_url($url.$source, $url, $path);
> +	if ( !$gs ) {
> +		warn "Couldn't find revmap for $url$source\n";
> +		next;
> +	}

As mentioned in the other thread, that 'next' should now be 'return'.

Sam

Re: Efficiency and correctness patches for git-svn mergeinfo support

[Andrew Myrick]

On Sun, Dec 20, 2009 at 1:07 PM, Sam Vilain <sam@vilain.net> wrote:
> On Sat, 2009-12-19 at 14:15 -0800, Andrew Myrick wrote:
>> I tried cloning from a fairly recent revision that I knew was after
>> our switchover to svn 1.5, and I received a number of these errors:
>>
>>    Couldn't find revmap for [branch]
>>    Exiting subroutine via next at /Users/adm/libexec/git-core/git-svn line 2983.
>>    Exiting subroutine via next at /Users/adm/libexec/git-core/git-svn line 2983.
>>    Exiting subroutine via next at /Users/adm/libexec/git-core/git-svn line 2983.
>>
>> I'm not sure if this is expected, since I didn't clone from the whole
>> repo, but it did cause a lot of spew.  I'm starting a fresh clone now,
>> but it takes a few days to get through the whole repository.  I'm
>> fairly new to git, so I would welcome any tips on how I can test this
>> more quickly.
>
> Whoops, no, not expected, I'll post a minor correction.  That means that
> the branch which was merged in does not have git-svn metadata; ie, it's
> not being tracked explicitly.  If people are doing merging of things
> which aren't roots of branches you would expect this.  SVN, like
> Perforce, supports a confusing amount of flexibility in its merge
> tracking.  If [branch] is a real branch, then you'll want to see why it
> doesn't have metadata yet.  Is it really a sub-tree of a real branch?
> You could fetch it independently using a separate git-svn remote, or you
> could ignore the warning; it should be relatively self-evident what
> happened from the merge message and the contents of the changeset.

I think the problem is that I started fetching from a later revision
than when the branch was created and reintegrated.  There would be no
metadata for the branch, so a lookup for it would fail.  Fetching the
branch explicitly, or fetching from r1 as my current test is doing,
should not experience this problem.

> Note if your repository was significantly re-organized at any point, it
> will pay to treat each section of history as a separate import project,
> and stitch the results together afterwards using grafts and
> filter-branch.

I don't believe it was, but I will keep that in mind if I run into
trouble.  The import from r1 went fine with git v1.6.5.*, and it's
proceeding well now.

> This version should be *significantly* faster than the old one.  ie, it
> should not take a minute per commit while importing the heavily
> merged-into integration branch.  Possibly a few seconds at most.

So far, most commits are fetched in under 3s with your new version.

Thanks for your help, Sam.

-Andrew

Re: [PATCH 2/5] git-svn: memoize conversion of SVN merge ticket info to git commit ranges

[Eric Wong]

Sam Vilain <sam@vilain.net> wrote:
> On Sun, 2009-12-20 at 05:33 +1300, Sam Vilain wrote:
> > +sub lookup_svn_merge {
> > +	my $uuid = shift;
> > +	my $url = shift;
> > +	my $merge = shift;
> > +
> > +	my ($source, $revs) = split ":", $merge;
> > +	my $path = $source;
> > +	$path =~ s{^/}{};
> > +	my $gs = Git::SVN->find_by_url($url.$source, $url, $path);
> > +	if ( !$gs ) {
> > +		warn "Couldn't find revmap for $url$source\n";
> > +		next;
> > +	}
> 
> As mentioned in the other thread, that 'next' should now be 'return'.

Thanks Sam and Andrew.  I've acked this series and pushed them out along
with a release notes update as well as another small fix (inline below).

Eric Wong (2):
      git svn: fix --revision when fetching deleted paths
      update release notes for git svn in 1.6.6

Sam Vilain (5):
      git-svn: expand the svn mergeinfo test suite, highlighting some failures
      git-svn: memoize conversion of SVN merge ticket info to git commit ranges
      git-svn: fix some mistakes with interpreting SVN mergeinfo commit ranges
      git-svn: exclude already merged tips using one rev-list call
      git-svn: detect cherry-picks correctly.

>From 577e9fcad2c8968846b365226b89778050496a78 Mon Sep 17 00:00:00 2001
From: Eric Wong <normalperson@yhbt.net>
Date: Mon, 21 Dec 2009 02:06:04 -0800
Subject: [PATCH] git svn: fix --revision when fetching deleted paths

When using the -r/--revision argument to fetch deleted history,
calling SVN::Ra::get_log() from an SVN::Ra object initialized
to track the deleted URL will fail.

This regression was introduced in:
  commit 4aacaeb3dc82bb6479e70e120053dc27a399460e
  "fix shallow clone when upstream revision is too new"

We now ignore errors from SVN::Ra::get_log() here because using
--revision will always override the value of $head here if
(and only if) we're tracking deleted directories.

Signed-off-by: Eric Wong <normalperson@yhbt.net>
---
 git-svn.perl |    6 +++++-
 1 files changed, 5 insertions(+), 1 deletions(-)

diff --git a/git-svn.perl b/git-svn.perl
index d362de7..a6f5061 100755
--- a/git-svn.perl
+++ b/git-svn.perl
@@ -1741,7 +1741,11 @@ sub fetch_all {
 	my $ra = Git::SVN::Ra->new($url);
 	my $uuid = $ra->get_uuid;
 	my $head = $ra->get_latest_revnum;
-	$ra->get_log("", $head, 0, 1, 0, 1, sub { $head = $_[1] });
+
+	# ignore errors, $head revision may not even exist anymore
+	eval { $ra->get_log("", $head, 0, 1, 0, 1, sub { $head = $_[1] }) };
+	warn "W: $@\n" if $@;
+
 	my $base = defined $fetch ? $head : 0;
 
 	# read the max revs for wildcard expansion (branches/*, tags/*)
-- 
Eric Wong