git.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* [GSoC'11] [PATCH/RFC v2 0/9] remote-svn-alpha updates
@ 2011-07-13 15:26 Dmitry Ivankov
  2011-07-13 15:26 ` [PATCH v2 1/9] svn-fe: use svnrdump --quiet in remote-svn-alpha Dmitry Ivankov
                   ` (8 more replies)
  0 siblings, 9 replies; 10+ messages in thread
From: Dmitry Ivankov @ 2011-07-13 15:26 UTC (permalink / raw)
  To: git; +Cc: Jonathan Nieder, David Barr, Ramkumar Ramachandra, Dmitry Ivankov

This is a second iteration of remote-svn-alpha series [1].

The main changes are:
1) more strict errors checking - be ready for helper exit code being checked[3]
2) saving "rN" marks as a commit->svn revision mapping
3) supporting incremental imports

"2)" could be even nicer if there were a way for git-remote add or git-clone to
setup a show-notes config option. Another interesting aspect is notes_ref commits
author (vcs-svn <vcs-svn@local>) and timestamps ("now"). Timestamps make notes sha1
diverge on different repos/importers, so fetching notes from a peer git repo is not
fast-forward and a bad thing is they are stored in a private namespace - not likely
place for user to perform merges. But as the tests show, one may want to trust peer
notes and fetch --force svn updated imported by a peer.

But anyway, there are a few tests in this series and it looks like the helper is
really able to import from svn incrementally and exchange/clone imported data
between different git repositories. Same limitations as earlier: only whole svnroot,
only import, needs svnrdump (for file:// svnadmin+svnlook would suffice)

Further improvements for a just import from svn and maybe clone/fetch svn data from
another git peers are:
a) mentioned above "2)" problems
b) adding various configuration options (like svn username and password)
c) add progress indication, either true percentage of # of imported revisions or
some adaptive indicator or any other informative one
d) maybe issue a checkpoint command from time to time or somehow allow to specify
how many revisions to fetch - for huge initial imports
e) import a subdirectory (of svnroot) history
f) shallow imports
g) save all revprops/fileprops whether they have git counterparts or not, ideally
these are sufficient to convert the imported history back to svn lossless
h) get rid of bashisms in this helper or rewrite it in C.

These do not block proceeding to the next topics:
i) push commits to svn
ii) track svn branches.


The patch base is svn-fe-pu at git://repo.or.cz/git/jrn.git
also a commit 7153183171de77d084a4c24ef19d23d6313ded2a can be used
as an earlier patch base.

For the last five commits (those just after the "add a test") some 
new svn-fe options[2] are required.

Whole git-remote-svn-alpha and a test script are cited at the bottom
of this letter as a quick reference for those who don't track svn-fe-pu.

[1] http://thread.gmane.org/gmane.comp.version-control.git/176617
[2] http://thread.gmane.org/gmane.comp.version-control.git/177025
[3] http://thread.gmane.org/gmane.comp.version-control.git/176002/focus=176019

Dmitry Ivankov (9):
  svn-fe: use svnrdump --quiet in remote-svn-alpha
  svn-fe: avoid error on no-op imports in remote-svn-alpha
  svn-fe: allow svnadmin instead of svnrdump in remote-svn-alpha
  svn-fe: add a test for remote-svn-alpha
  svn-fe: use svn-fe --no-progress in remote-svn-alpha
  svn-fe: use proper refspec in remote-svn-alpha
  svn-fe: write svnrev notes in remote-svn-alpha
  svn-fe: import incrementally in svn-remote-alpha
  svn-fe: reuse import-marks in remote-svn-alpha

 contrib/svn-fe/git-remote-svn-alpha        |  140 +++++++++++-
 contrib/svn-fe/t/.gitignore                |    3 +
 contrib/svn-fe/t/t9010-remote-svn-alpha.sh |  351 ++++++++++++++++++++++++++++
 3 files changed, 485 insertions(+), 9 deletions(-)
 create mode 100644 contrib/svn-fe/t/.gitignore
 create mode 100755 contrib/svn-fe/t/t9010-remote-svn-alpha.sh

-- 
1.7.3.4
diff --git a/contrib/svn-fe/git-remote-svn-alpha b/contrib/svn-fe/git-remote-svn-alpha
new file mode 100755
index 0000000..8096ef2
--- /dev/null
+++ b/contrib/svn-fe/git-remote-svn-alpha
@@ -0,0 +1,177 @@
+#!/bin/bash
+set -o pipefail
+set -e
+
+die () {
+	printf >&2 'fatal: %s\n' "$*"
+	exit 128
+}
+
+usage () {
+	printf >&2 'usage: %s\n' "$*"
+	exit 129
+}
+
+svnrdump_wrap () {
+	exec 6<&1 &&
+
+	EX=$(
+		(svnrdump dump --non-interactive --username=Guest --password= \
+			--quiet "$1" "$2" >&6) 2>&1; test $? -ne 0 || echo Success
+	) &&
+	if test "z$EX" != "zSuccess"; then
+		if test "z$EX" = "zLOWER cannot be greater than UPPER."; then
+			return 0
+		fi
+		echo "$EX" >&2
+		return 1
+	fi
+}
+
+try_svnrdump () {
+	command -v svnrdump >/dev/null &&
+	echo svnrdump_wrap ||
+	true
+}
+
+svnadmin_wrap () {
+	path=${1##file://} &&
+	test "z$path" != "z$1" &&
+	latest=$(svnlook youngest "$path") &&
+	future=$(( $latest + 1 )) &&
+	asked=${2#-r} &&
+	asked=${asked%:*} &&
+	test "$asked" -eq "$future" ||
+	svnadmin dump --incremental --deltas --quiet "$path" "$2"
+}
+
+try_svnadmin () {
+	command -v svnadmin >/dev/null &&
+	command -v svnlook >/dev/null &&
+	echo svnadmin_wrap ||
+	true
+}
+
+SVNDUMP=""
+SVNDUMP=${SVNDUMP:-`try_svnrdump`}
+SVNDUMP=${SVNDUMP:-`try_svnadmin`}
+test -n "$SVNDUMP" || die "neither svnrdump nor svnadmin & svnlook was found"
+
+git_dir=""
+marks_dir="info/fast-import/svn-alpha/$repo"
+
+do_gen_marks () {
+	notes=$1
+	test "z$git_dir" != "$z" || die "we were not told the gitdir"
+	git --git-dir="$git_dir" ls-tree -r "$notes" |
+	{
+		while read -r mode type sha path
+		do
+			data=$(git --git-dir="$git_dir" cat-file blob $sha | tail -n 1)
+			data=${data##r}
+			commit=$(echo $path | tr -d /)
+			echo ":$data $commit"
+		done
+	}
+}
+
+prepare_marks () {
+	dst=$1 notes=$2 rev=$3
+	test "z$git_dir" != "$z" || die "we were not told the gitdir"
+	path="$git_dir/$marks_dir"
+	mkdir -p "$path"
+	path="$path/marks"
+
+	if test ! -f "$path"; then
+		touch "$path"
+	fi
+	if test "$rev" = "-1"; then
+		return 0
+	fi
+	mark_sha=""
+	{
+		while read -r m sha
+		do
+			if test "$m" = ":$rev"; then
+				mark_sha="$sha"
+			fi
+		done
+	} <"$path"
+	dst_sha=$( git rev-parse "$dst" )
+	if test -n "$mark_sha"; then
+		test "$mark_sha" = "$dst_sha" || die "latest mark and note diverge"
+	else
+		do_gen_marks "$notes" >"$path"
+	fi
+}
+
+last_imported_rev () {
+	dst=$1 notes=$2
+	test "z$git_dir" != "$z" || die "we were not told the gitdir"
+	git --git-dir="$git_dir" show-ref -q --verify $dst || {
+		echo "-1"
+		return 0
+	}
+	rev=$(git --git-dir="$git_dir" log --show-notes="$notes" -1 --format=%N $dst)
+	rev=${rev##r}
+	test "z$rev" != "z" || {
+		die "remote HEAD has no note"
+	}
+	echo $rev
+}
+
+do_import () {
+	revs=$1 url=$2 dst=$3 notes=$4
+	rev=$(last_imported_rev "$dst" "$notes")
+	start_rev=$(($rev + 1))
+	revs="$start_rev:HEAD"
+	prepare_marks $dst $notes $rev
+	echo "feature import-marks=$git_dir/$marks_dir/marks"
+	echo "feature export-marks=$git_dir/$marks_dir/marks"
+	(eval "$SVNDUMP \"$url\" -r\"$revs\"" |	svn-fe --ref="$dst" --notes-ref="$notes" --incremental --no-progress) 3<&0 || die "FAILURE"
+	exec 1>&-
+}
+
+test "${2+set}" ||
+usage 'git remote-svn-alpha <repository> <URL> < commandlist'
+repo=$1
+url=$2
+need_import=""
+remote_ref="refs/heads/master"
+private_ref="refs/svn-alpha/$repo/SVNHEAD"
+remote_notes="refs/notes/svnr"
+private_notes="refs/svn-alpha/$repo/SVNR"
+
+while read -r cmd args
+do
+	case $cmd in
+	capabilities)
+		echo import
+		echo gitdir
+		echo "refspec HEAD:$private_ref"
+		echo "refspec $remote_ref:$private_ref"
+		echo "refspec $remote_notes:$private_notes"
+		echo
+		;;
+	gitdir)
+		git_dir="$args"
+		;;
+	list)
+		echo "? HEAD"
+		echo "? $remote_ref"
+		echo
+		;;
+	import)
+		test "$args" = "HEAD" || test "$args" = "$remote_ref" ||
+		die "remote-svn-alpha: unsupported import ref argument: $args"
+		need_import="yes"
+		;;
+	'')
+		test "$need_import" = "yes" || exit 0
+		do_import 0:HEAD "$url" "$private_ref" "$private_notes"
+		need_import=""
+		;;
+	*)
+		die "remote-svn-alpha: unsupported command: $cmd $args"
+	esac
+done
diff --git a/contrib/svn-fe/t/t9010-remote-svn-alpha.sh b/contrib/svn-fe/t/t9010-remote-svn-alpha.sh
new file mode 100755
index 0000000..9d9aca0
--- /dev/null
+++ b/contrib/svn-fe/t/t9010-remote-svn-alpha.sh
@@ -0,0 +1,351 @@
+#!/bin/sh
+
+test_description='check svn-alpha remote helper'
+
+PATH=$(pwd)/..:$PATH
+TEST_DIRECTORY=$(pwd)/../../../t
+. $TEST_DIRECTORY/test-lib.sh
+
+if command -v svnrdump >/dev/null; then
+	test_set_prereq SVNRDUMP
+fi
+
+deinit_git () {
+	rm -fr .git
+}
+
+reinit_git () {
+	deinit_git &&
+	git init
+}
+
+properties () {
+	while test "$#" -ne 0
+	do
+		property="$1" &&
+		value="$2" &&
+		printf "%s\n" "K ${#property}" &&
+		printf "%s\n" "$property" &&
+		printf "%s\n" "V ${#value}" &&
+		printf "%s\n" "$value" &&
+		shift 2 ||
+		return 1
+	done
+}
+
+text_no_props () {
+	text="$1
+" &&
+	printf "%s\n" "Prop-content-length: 10" &&
+	printf "%s\n" "Text-content-length: ${#text}" &&
+	printf "%s\n" "Content-length: $((${#text} + 10))" &&
+	printf "%s\n" "" "PROPS-END" &&
+	printf "%s\n" "$text"
+}
+
+dump_to_svnrepo () {
+	dump="$1" &&
+	path="$2" &&
+	svnadmin create "$path" &&
+	svnadmin load "$path" < "$dump"
+}
+
+svnurl () {
+	printf "svn-alpha::file://%s/%s" "$(pwd)" "$1"
+}
+
+test_nr_revs () {
+	n=$1 &&
+	repo=$2 &&
+	ref=$3 &&
+	git --git-dir="$repo" log --format=oneline "$ref" >revs &&
+	test_line_count = "$n" revs
+}
+
+test_expect_success 'svnadmin is present' '
+	command -v svnadmin &&
+	test_set_prereq SVNADMIN
+'
+
+test_expect_success SVNADMIN 'create empty svnrepo' '
+	echo "SVN-fs-dump-format-version: 2" > empty.dump &&
+	dump_to_svnrepo empty.dump empty.svn &&
+	test_set_prereq EMPTY_SVN
+'
+
+test_expect_success SVNADMIN 'create tiny svnrepo' '
+	{
+		properties \
+			svn:author author@example.com \
+			svn:date "1999-02-01T00:01:002.000000Z" \
+			svn:log "add directory with some files in it" &&
+		echo PROPS-END
+	} >props &&
+	{
+		cat <<-EOF &&
+		SVN-fs-dump-format-version: 3
+
+		Revision-number: 1
+		EOF
+		echo Prop-content-length: $(wc -c <props) &&
+		echo Content-length: $(wc -c <props) &&
+		echo &&
+		cat props &&
+		cat <<-\EOF &&
+
+		Node-path: directory
+		Node-kind: dir
+		Node-action: add
+		Prop-content-length: 10
+		Content-length: 10
+
+		PROPS-END
+		Node-path: directory/somefile
+		Node-kind: file
+		Node-action: add
+		EOF
+		text_no_props hi
+	} >tiny.dump &&
+	dump_to_svnrepo tiny.dump tiny.svn &&
+	test_set_prereq TINY_SVN
+'
+
+test_expect_success SVNADMIN 'create small svndump' '
+	{
+		properties \
+			svn:author author@example.com \
+			svn:date "1999-02-01T00:01:002.000000Z" \
+			svn:log "add directory with some files in it" &&
+		echo PROPS-END
+	} >props &&
+	{
+		echo Prop-content-length: $(wc -c <props) &&
+		echo Content-length: $(wc -c <props) &&
+		echo &&
+		cat props
+	} >props_dump &&
+	cat >small.dump.r0 <<-EOF &&
+	SVN-fs-dump-format-version: 3
+	EOF
+	for x in `seq 1 1 10`; do
+		{
+			echo &&
+			echo "Revision-number: $x" &&
+			cat props_dump &&
+
+			if test "$x" -eq "1"; then
+				cat <<-\EOF
+
+				Node-path: directory
+				Node-kind: dir
+				Node-action: add
+				Prop-content-length: 10
+				Content-length: 10
+
+				PROPS-END
+
+				EOF
+			fi
+			echo "Node-path: directory/somefile$x" &&
+			echo "Node-kind: file" &&
+			echo "Node-action: add" &&
+			text_no_props hi
+		} >small.dump.r$x
+	done &&
+	test_set_prereq SMALL_SVNDUMP
+'
+
+test_expect_success SMALL_SVNDUMP 'create small svnrepo' '
+	rm -rf small.svn* &&
+	svnadmin create small.svn &&
+	for x in `seq 1 1 10`; do
+		cat small.dump.r0 small.dump.r$x >part.dump &&
+		svnadmin load small.svn <part.dump &&
+		cp -r small.svn small.svn.r0-$x
+	done &&
+	test_set_prereq SMALL_SVN
+'
+
+test_expect_failure EMPTY_SVN 'fetch empty' '
+	reinit_git &&
+	url=$(svnurl empty.svn) &&
+	git remote add svn "$url" &&
+	git fetch svn
+'
+
+test_expect_failure TINY_SVN 'clone tiny' '
+	deinit_git &&
+	url=$(svnurl tiny.svn) &&
+	git clone "$url" tiny1.git &&
+	test_nr_revs 1 tiny1.git refs/remotes/origin/master
+'
+
+test_expect_success TINY_SVN 'clone --mirror tiny' '
+	deinit_git &&
+	url=$(svnurl tiny.svn) &&
+	git clone --mirror "$url" tiny2.git &&
+	test_nr_revs 1 tiny2.git refs/heads/master
+'
+
+test_expect_success TINY_SVN 'clone --bare tiny' '
+	deinit_git &&
+	url=$(svnurl tiny.svn) &&
+	git clone --bare "$url" tiny3.git &&
+	test_nr_revs 1 tiny3.git refs/heads/master
+'
+
+test_expect_success TINY_SVN 'clone -b master tiny' '
+	deinit_git &&
+	url=$(svnurl tiny.svn) &&
+	git clone -b master "$url" tiny4.git &&
+	test_nr_revs 1 tiny4.git/.git refs/heads/master
+'
+
+test_expect_success SMALL_SVN 'clone -b master small' '
+	deinit_git &&
+	url=$(svnurl small.svn) &&
+	git clone -b master "$url" small.git &&
+	test_nr_revs 10 small.git/.git refs/heads/master
+'
+
+test_expect_success TINY_SVN,SVNRDUMP 'no crash on clone url/path' '
+	deinit_git &&
+	url=$(svnurl small.svn)/directory &&
+	git clone -b master "$url" small_dir.git &&
+	test_nr_revs 10 small_dir.git/.git refs/heads/master
+'
+
+test_expect_success TINY_SVN,SVNRDUMP 'no crash on clone url/path/file' '
+	deinit_git &&
+	url=$(svnurl small.svn)/directory/somefile3 &&
+	git clone -b master "$url" small_dir_file.git &&
+	test_nr_revs 10 small_dir_file.git/.git refs/heads/master
+'
+
+test_expect_success SMALL_SVN 'fetch each rev of SMALL separately' '
+	reinit_git &&
+	url=$(svnurl small.svn) &&
+
+	for x in `seq 1 1 10`; do
+		git remote add svn_$x "$url.r0-$x"
+	done &&
+	git remote update &&
+
+	git remote add svn "$url" &&
+	git fetch svn &&
+
+	git rev-parse -s remotes/svn_7/master~5 >ref7_2 &&
+	git rev-parse -s remotes/svn/master~8 >ref_2 &&
+	test_cmp ref7_2 ref_2
+'
+
+test_expect_success SMALL_SVN 'fetch updates from SMALL' '
+	reinit_git &&
+	url=$(svnurl link.svn) &&
+	git remote add svn "$url" &&
+
+	ln -sfn small.svn.r0-5 link.svn &&
+	git fetch svn &&
+	test_nr_revs 5 .git refs/remotes/svn/master &&
+
+	ln -sfn small.svn.r0-10 link.svn &&
+	git fetch svn &&
+	test_nr_revs 10 .git refs/remotes/svn/master &&
+
+	git fetch svn &&
+	test_nr_revs 10 .git refs/remotes/svn/master
+'
+
+test_expect_success TINY_SVN 'fetch TINY does not write to refs/heads/master' '
+	reinit_git &&
+	url=$(svnurl tiny.svn) &&
+	git remote add svn "$url" &&
+	git fetch svn &&
+	git show-ref --verify refs/remotes/svn/master &&
+	test_must_fail git show-ref --verify refs/heads/master
+'
+
+test_expect_success SMALL_SVN 'fetch SMALL writes revnum notes' '
+	reinit_git &&
+	url=$(svnurl small.svn) &&
+	git remote add svn "$url" &&
+	git fetch svn &&
+	git log --show-notes=refs/svn-alpha/svn/SVNR --format=%N -1 refs/remotes/svn/master^ >actual.note &&
+	echo r9 >expect.note &&
+	echo >>expect.note &&
+	test_cmp expect.note actual.note
+'
+
+test_expect_success SMALL_SVN 'marks from notes regeneration' '
+	reinit_git &&
+	url=$(svnurl link.svn) &&
+	git remote add svn "$url" &&
+
+	ln -sfn small.svn.r0-5 link.svn &&
+	git fetch svn &&
+	test_nr_revs 5 .git refs/remotes/svn/master &&
+
+	rm -rf .git/info/fast-import/svn-alpha &&
+
+	ln -sfn small.svn.r0-10 link.svn &&
+	git fetch svn &&
+	test_nr_revs 10 .git refs/remotes/svn/master
+'
+
+test_expect_success SMALL_SVN 'clone to bootstrap' '
+	deinit_git &&
+	url=$(svnurl link.svn) &&
+	ln -sfn small.svn.r0-5 link.svn &&
+	git clone -b master "$url" master.git &&
+	{
+		cd master.git &&
+		git branch pub_head refs/svn-alpha/origin/SVNHEAD &&
+		git branch pub_notes refs/svn-alpha/origin/SVNR &&
+		cd ..
+	} &&
+	ln -sfn small.svn.r0-8 link.svn &&
+	git clone master.git slave.git &&
+	{
+		cd slave.git &&
+		git remote add svn "$url" &&
+		git update-ref refs/svn-alpha/svn/SVNHEAD refs/remotes/origin/pub_head &&
+		git update-ref refs/svn-alpha/svn/SVNR refs/remotes/origin/pub_notes &&
+		git fetch svn &&
+		git merge-base refs/svn-alpha/svn/SVNR refs/remotes/origin/pub_notes &&
+		cd ..
+	} &&
+	test_nr_revs 8 slave.git/.git refs/remotes/svn/master
+'
+
+test_expect_success SMALL_SVN 'clone and exchange' '
+	deinit_git &&
+	url=$(svnurl link.svn) &&
+	ln -sfn small.svn.r0-1 link.svn &&
+	git clone -b master "$url" A.git &&
+	git clone -b master "$url" B.git &&
+	test_nr_revs 1 A.git/.git origin/master &&
+	test_nr_revs 1 B.git/.git origin/master &&
+
+	git --git-dir=A.git/.git remote add B B.git &&
+	git --git-dir=B.git/.git remote add A A.git &&
+	refspecs="+refs/svn-alpha/origin/SVNR:refs/svn-alpha/origin/SVNR"
+	refspecs="$refspecs refs/svn-alpha/origin/SVNHEAD:refs/svn-alpha/origin/SVNHEAD"
+
+	ln -sfn small.svn.r0-2 link.svn &&
+	git --git-dir=A.git/.git fetch origin &&
+
+	git --git-dir=B.git/.git fetch -f A $refspecs &&
+
+	ln -sfn small.svn.r0-3 link.svn &&
+	git --git-dir=B.git/.git fetch origin &&
+
+	git --git-dir=A.git/.git fetch B $refspecs &&
+
+	git --git-dir=A.git/.git fetch origin &&
+	git --git-dir=B.git/.git fetch origin &&
+
+	test_nr_revs 3 A.git/.git origin/master &&
+	test_nr_revs 3 B.git/.git origin/master
+'
+
+test_done

^ permalink raw reply related	[flat|nested] 10+ messages in thread
end of thread, other threads:[~2011-07-13 15:27 UTC | newest]

Thread overview: 10+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2011-07-13 15:26 [GSoC'11] [PATCH/RFC v2 0/9] remote-svn-alpha updates Dmitry Ivankov
2011-07-13 15:26 ` [PATCH v2 1/9] svn-fe: use svnrdump --quiet in remote-svn-alpha Dmitry Ivankov
2011-07-13 15:26 ` [PATCH v2 2/9] svn-fe: avoid error on no-op imports " Dmitry Ivankov
2011-07-13 15:26 ` [PATCH v2 3/9] svn-fe: allow svnadmin instead of svnrdump " Dmitry Ivankov
2011-07-13 15:26 ` [PATCH v2 4/9] svn-fe: add a test for remote-svn-alpha Dmitry Ivankov
2011-07-13 15:26 ` [PATCH v2 5/9] svn-fe: use svn-fe --no-progress in remote-svn-alpha Dmitry Ivankov
2011-07-13 15:26 ` [PATCH v2 6/9] svn-fe: use proper refspec " Dmitry Ivankov
2011-07-13 15:26 ` [PATCH v2 7/9] svn-fe: write svnrev notes " Dmitry Ivankov
2011-07-13 15:26 ` [PATCH v2 8/9] svn-fe: import incrementally in svn-remote-alpha Dmitry Ivankov
2011-07-13 15:26 ` [PATCH v2 9/9] svn-fe: reuse import-marks in remote-svn-alpha Dmitry Ivankov

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).