* [GSoC'11] [PATCH/RFC v2 0/9] remote-svn-alpha updates
@ 2011-07-13 15:26 Dmitry Ivankov
2011-07-13 15:26 ` [PATCH v2 1/9] svn-fe: use svnrdump --quiet in remote-svn-alpha Dmitry Ivankov
` (8 more replies)
0 siblings, 9 replies; 10+ messages in thread
From: Dmitry Ivankov @ 2011-07-13 15:26 UTC (permalink / raw)
To: git; +Cc: Jonathan Nieder, David Barr, Ramkumar Ramachandra, Dmitry Ivankov
This is a second iteration of remote-svn-alpha series [1].
The main changes are:
1) more strict errors checking - be ready for helper exit code being checked[3]
2) saving "rN" marks as a commit->svn revision mapping
3) supporting incremental imports
"2)" could be even nicer if there were a way for git-remote add or git-clone to
setup a show-notes config option. Another interesting aspect is notes_ref commits
author (vcs-svn <vcs-svn@local>) and timestamps ("now"). Timestamps make notes sha1
diverge on different repos/importers, so fetching notes from a peer git repo is not
fast-forward and a bad thing is they are stored in a private namespace - not likely
place for user to perform merges. But as the tests show, one may want to trust peer
notes and fetch --force svn updated imported by a peer.
But anyway, there are a few tests in this series and it looks like the helper is
really able to import from svn incrementally and exchange/clone imported data
between different git repositories. Same limitations as earlier: only whole svnroot,
only import, needs svnrdump (for file:// svnadmin+svnlook would suffice)
Further improvements for a just import from svn and maybe clone/fetch svn data from
another git peers are:
a) mentioned above "2)" problems
b) adding various configuration options (like svn username and password)
c) add progress indication, either true percentage of # of imported revisions or
some adaptive indicator or any other informative one
d) maybe issue a checkpoint command from time to time or somehow allow to specify
how many revisions to fetch - for huge initial imports
e) import a subdirectory (of svnroot) history
f) shallow imports
g) save all revprops/fileprops whether they have git counterparts or not, ideally
these are sufficient to convert the imported history back to svn lossless
h) get rid of bashisms in this helper or rewrite it in C.
These do not block proceeding to the next topics:
i) push commits to svn
ii) track svn branches.
The patch base is svn-fe-pu at git://repo.or.cz/git/jrn.git
also a commit 7153183171de77d084a4c24ef19d23d6313ded2a can be used
as an earlier patch base.
For the last five commits (those just after the "add a test") some
new svn-fe options[2] are required.
Whole git-remote-svn-alpha and a test script are cited at the bottom
of this letter as a quick reference for those who don't track svn-fe-pu.
[1] http://thread.gmane.org/gmane.comp.version-control.git/176617
[2] http://thread.gmane.org/gmane.comp.version-control.git/177025
[3] http://thread.gmane.org/gmane.comp.version-control.git/176002/focus=176019
Dmitry Ivankov (9):
svn-fe: use svnrdump --quiet in remote-svn-alpha
svn-fe: avoid error on no-op imports in remote-svn-alpha
svn-fe: allow svnadmin instead of svnrdump in remote-svn-alpha
svn-fe: add a test for remote-svn-alpha
svn-fe: use svn-fe --no-progress in remote-svn-alpha
svn-fe: use proper refspec in remote-svn-alpha
svn-fe: write svnrev notes in remote-svn-alpha
svn-fe: import incrementally in svn-remote-alpha
svn-fe: reuse import-marks in remote-svn-alpha
contrib/svn-fe/git-remote-svn-alpha | 140 +++++++++++-
contrib/svn-fe/t/.gitignore | 3 +
contrib/svn-fe/t/t9010-remote-svn-alpha.sh | 351 ++++++++++++++++++++++++++++
3 files changed, 485 insertions(+), 9 deletions(-)
create mode 100644 contrib/svn-fe/t/.gitignore
create mode 100755 contrib/svn-fe/t/t9010-remote-svn-alpha.sh
--
1.7.3.4
diff --git a/contrib/svn-fe/git-remote-svn-alpha b/contrib/svn-fe/git-remote-svn-alpha
new file mode 100755
index 0000000..8096ef2
--- /dev/null
+++ b/contrib/svn-fe/git-remote-svn-alpha
@@ -0,0 +1,177 @@
+#!/bin/bash
+set -o pipefail
+set -e
+
+die () {
+ printf >&2 'fatal: %s\n' "$*"
+ exit 128
+}
+
+usage () {
+ printf >&2 'usage: %s\n' "$*"
+ exit 129
+}
+
+svnrdump_wrap () {
+ exec 6<&1 &&
+
+ EX=$(
+ (svnrdump dump --non-interactive --username=Guest --password= \
+ --quiet "$1" "$2" >&6) 2>&1; test $? -ne 0 || echo Success
+ ) &&
+ if test "z$EX" != "zSuccess"; then
+ if test "z$EX" = "zLOWER cannot be greater than UPPER."; then
+ return 0
+ fi
+ echo "$EX" >&2
+ return 1
+ fi
+}
+
+try_svnrdump () {
+ command -v svnrdump >/dev/null &&
+ echo svnrdump_wrap ||
+ true
+}
+
+svnadmin_wrap () {
+ path=${1##file://} &&
+ test "z$path" != "z$1" &&
+ latest=$(svnlook youngest "$path") &&
+ future=$(( $latest + 1 )) &&
+ asked=${2#-r} &&
+ asked=${asked%:*} &&
+ test "$asked" -eq "$future" ||
+ svnadmin dump --incremental --deltas --quiet "$path" "$2"
+}
+
+try_svnadmin () {
+ command -v svnadmin >/dev/null &&
+ command -v svnlook >/dev/null &&
+ echo svnadmin_wrap ||
+ true
+}
+
+SVNDUMP=""
+SVNDUMP=${SVNDUMP:-`try_svnrdump`}
+SVNDUMP=${SVNDUMP:-`try_svnadmin`}
+test -n "$SVNDUMP" || die "neither svnrdump nor svnadmin & svnlook was found"
+
+git_dir=""
+marks_dir="info/fast-import/svn-alpha/$repo"
+
+do_gen_marks () {
+ notes=$1
+ test "z$git_dir" != "$z" || die "we were not told the gitdir"
+ git --git-dir="$git_dir" ls-tree -r "$notes" |
+ {
+ while read -r mode type sha path
+ do
+ data=$(git --git-dir="$git_dir" cat-file blob $sha | tail -n 1)
+ data=${data##r}
+ commit=$(echo $path | tr -d /)
+ echo ":$data $commit"
+ done
+ }
+}
+
+prepare_marks () {
+ dst=$1 notes=$2 rev=$3
+ test "z$git_dir" != "$z" || die "we were not told the gitdir"
+ path="$git_dir/$marks_dir"
+ mkdir -p "$path"
+ path="$path/marks"
+
+ if test ! -f "$path"; then
+ touch "$path"
+ fi
+ if test "$rev" = "-1"; then
+ return 0
+ fi
+ mark_sha=""
+ {
+ while read -r m sha
+ do
+ if test "$m" = ":$rev"; then
+ mark_sha="$sha"
+ fi
+ done
+ } <"$path"
+ dst_sha=$( git rev-parse "$dst" )
+ if test -n "$mark_sha"; then
+ test "$mark_sha" = "$dst_sha" || die "latest mark and note diverge"
+ else
+ do_gen_marks "$notes" >"$path"
+ fi
+}
+
+last_imported_rev () {
+ dst=$1 notes=$2
+ test "z$git_dir" != "$z" || die "we were not told the gitdir"
+ git --git-dir="$git_dir" show-ref -q --verify $dst || {
+ echo "-1"
+ return 0
+ }
+ rev=$(git --git-dir="$git_dir" log --show-notes="$notes" -1 --format=%N $dst)
+ rev=${rev##r}
+ test "z$rev" != "z" || {
+ die "remote HEAD has no note"
+ }
+ echo $rev
+}
+
+do_import () {
+ revs=$1 url=$2 dst=$3 notes=$4
+ rev=$(last_imported_rev "$dst" "$notes")
+ start_rev=$(($rev + 1))
+ revs="$start_rev:HEAD"
+ prepare_marks $dst $notes $rev
+ echo "feature import-marks=$git_dir/$marks_dir/marks"
+ echo "feature export-marks=$git_dir/$marks_dir/marks"
+ (eval "$SVNDUMP \"$url\" -r\"$revs\"" | svn-fe --ref="$dst" --notes-ref="$notes" --incremental --no-progress) 3<&0 || die "FAILURE"
+ exec 1>&-
+}
+
+test "${2+set}" ||
+usage 'git remote-svn-alpha <repository> <URL> < commandlist'
+repo=$1
+url=$2
+need_import=""
+remote_ref="refs/heads/master"
+private_ref="refs/svn-alpha/$repo/SVNHEAD"
+remote_notes="refs/notes/svnr"
+private_notes="refs/svn-alpha/$repo/SVNR"
+
+while read -r cmd args
+do
+ case $cmd in
+ capabilities)
+ echo import
+ echo gitdir
+ echo "refspec HEAD:$private_ref"
+ echo "refspec $remote_ref:$private_ref"
+ echo "refspec $remote_notes:$private_notes"
+ echo
+ ;;
+ gitdir)
+ git_dir="$args"
+ ;;
+ list)
+ echo "? HEAD"
+ echo "? $remote_ref"
+ echo
+ ;;
+ import)
+ test "$args" = "HEAD" || test "$args" = "$remote_ref" ||
+ die "remote-svn-alpha: unsupported import ref argument: $args"
+ need_import="yes"
+ ;;
+ '')
+ test "$need_import" = "yes" || exit 0
+ do_import 0:HEAD "$url" "$private_ref" "$private_notes"
+ need_import=""
+ ;;
+ *)
+ die "remote-svn-alpha: unsupported command: $cmd $args"
+ esac
+done
diff --git a/contrib/svn-fe/t/t9010-remote-svn-alpha.sh b/contrib/svn-fe/t/t9010-remote-svn-alpha.sh
new file mode 100755
index 0000000..9d9aca0
--- /dev/null
+++ b/contrib/svn-fe/t/t9010-remote-svn-alpha.sh
@@ -0,0 +1,351 @@
+#!/bin/sh
+
+test_description='check svn-alpha remote helper'
+
+PATH=$(pwd)/..:$PATH
+TEST_DIRECTORY=$(pwd)/../../../t
+. $TEST_DIRECTORY/test-lib.sh
+
+if command -v svnrdump >/dev/null; then
+ test_set_prereq SVNRDUMP
+fi
+
+deinit_git () {
+ rm -fr .git
+}
+
+reinit_git () {
+ deinit_git &&
+ git init
+}
+
+properties () {
+ while test "$#" -ne 0
+ do
+ property="$1" &&
+ value="$2" &&
+ printf "%s\n" "K ${#property}" &&
+ printf "%s\n" "$property" &&
+ printf "%s\n" "V ${#value}" &&
+ printf "%s\n" "$value" &&
+ shift 2 ||
+ return 1
+ done
+}
+
+text_no_props () {
+ text="$1
+" &&
+ printf "%s\n" "Prop-content-length: 10" &&
+ printf "%s\n" "Text-content-length: ${#text}" &&
+ printf "%s\n" "Content-length: $((${#text} + 10))" &&
+ printf "%s\n" "" "PROPS-END" &&
+ printf "%s\n" "$text"
+}
+
+dump_to_svnrepo () {
+ dump="$1" &&
+ path="$2" &&
+ svnadmin create "$path" &&
+ svnadmin load "$path" < "$dump"
+}
+
+svnurl () {
+ printf "svn-alpha::file://%s/%s" "$(pwd)" "$1"
+}
+
+test_nr_revs () {
+ n=$1 &&
+ repo=$2 &&
+ ref=$3 &&
+ git --git-dir="$repo" log --format=oneline "$ref" >revs &&
+ test_line_count = "$n" revs
+}
+
+test_expect_success 'svnadmin is present' '
+ command -v svnadmin &&
+ test_set_prereq SVNADMIN
+'
+
+test_expect_success SVNADMIN 'create empty svnrepo' '
+ echo "SVN-fs-dump-format-version: 2" > empty.dump &&
+ dump_to_svnrepo empty.dump empty.svn &&
+ test_set_prereq EMPTY_SVN
+'
+
+test_expect_success SVNADMIN 'create tiny svnrepo' '
+ {
+ properties \
+ svn:author author@example.com \
+ svn:date "1999-02-01T00:01:002.000000Z" \
+ svn:log "add directory with some files in it" &&
+ echo PROPS-END
+ } >props &&
+ {
+ cat <<-EOF &&
+ SVN-fs-dump-format-version: 3
+
+ Revision-number: 1
+ EOF
+ echo Prop-content-length: $(wc -c <props) &&
+ echo Content-length: $(wc -c <props) &&
+ echo &&
+ cat props &&
+ cat <<-\EOF &&
+
+ Node-path: directory
+ Node-kind: dir
+ Node-action: add
+ Prop-content-length: 10
+ Content-length: 10
+
+ PROPS-END
+ Node-path: directory/somefile
+ Node-kind: file
+ Node-action: add
+ EOF
+ text_no_props hi
+ } >tiny.dump &&
+ dump_to_svnrepo tiny.dump tiny.svn &&
+ test_set_prereq TINY_SVN
+'
+
+test_expect_success SVNADMIN 'create small svndump' '
+ {
+ properties \
+ svn:author author@example.com \
+ svn:date "1999-02-01T00:01:002.000000Z" \
+ svn:log "add directory with some files in it" &&
+ echo PROPS-END
+ } >props &&
+ {
+ echo Prop-content-length: $(wc -c <props) &&
+ echo Content-length: $(wc -c <props) &&
+ echo &&
+ cat props
+ } >props_dump &&
+ cat >small.dump.r0 <<-EOF &&
+ SVN-fs-dump-format-version: 3
+ EOF
+ for x in `seq 1 1 10`; do
+ {
+ echo &&
+ echo "Revision-number: $x" &&
+ cat props_dump &&
+
+ if test "$x" -eq "1"; then
+ cat <<-\EOF
+
+ Node-path: directory
+ Node-kind: dir
+ Node-action: add
+ Prop-content-length: 10
+ Content-length: 10
+
+ PROPS-END
+
+ EOF
+ fi
+ echo "Node-path: directory/somefile$x" &&
+ echo "Node-kind: file" &&
+ echo "Node-action: add" &&
+ text_no_props hi
+ } >small.dump.r$x
+ done &&
+ test_set_prereq SMALL_SVNDUMP
+'
+
+test_expect_success SMALL_SVNDUMP 'create small svnrepo' '
+ rm -rf small.svn* &&
+ svnadmin create small.svn &&
+ for x in `seq 1 1 10`; do
+ cat small.dump.r0 small.dump.r$x >part.dump &&
+ svnadmin load small.svn <part.dump &&
+ cp -r small.svn small.svn.r0-$x
+ done &&
+ test_set_prereq SMALL_SVN
+'
+
+test_expect_failure EMPTY_SVN 'fetch empty' '
+ reinit_git &&
+ url=$(svnurl empty.svn) &&
+ git remote add svn "$url" &&
+ git fetch svn
+'
+
+test_expect_failure TINY_SVN 'clone tiny' '
+ deinit_git &&
+ url=$(svnurl tiny.svn) &&
+ git clone "$url" tiny1.git &&
+ test_nr_revs 1 tiny1.git refs/remotes/origin/master
+'
+
+test_expect_success TINY_SVN 'clone --mirror tiny' '
+ deinit_git &&
+ url=$(svnurl tiny.svn) &&
+ git clone --mirror "$url" tiny2.git &&
+ test_nr_revs 1 tiny2.git refs/heads/master
+'
+
+test_expect_success TINY_SVN 'clone --bare tiny' '
+ deinit_git &&
+ url=$(svnurl tiny.svn) &&
+ git clone --bare "$url" tiny3.git &&
+ test_nr_revs 1 tiny3.git refs/heads/master
+'
+
+test_expect_success TINY_SVN 'clone -b master tiny' '
+ deinit_git &&
+ url=$(svnurl tiny.svn) &&
+ git clone -b master "$url" tiny4.git &&
+ test_nr_revs 1 tiny4.git/.git refs/heads/master
+'
+
+test_expect_success SMALL_SVN 'clone -b master small' '
+ deinit_git &&
+ url=$(svnurl small.svn) &&
+ git clone -b master "$url" small.git &&
+ test_nr_revs 10 small.git/.git refs/heads/master
+'
+
+test_expect_success TINY_SVN,SVNRDUMP 'no crash on clone url/path' '
+ deinit_git &&
+ url=$(svnurl small.svn)/directory &&
+ git clone -b master "$url" small_dir.git &&
+ test_nr_revs 10 small_dir.git/.git refs/heads/master
+'
+
+test_expect_success TINY_SVN,SVNRDUMP 'no crash on clone url/path/file' '
+ deinit_git &&
+ url=$(svnurl small.svn)/directory/somefile3 &&
+ git clone -b master "$url" small_dir_file.git &&
+ test_nr_revs 10 small_dir_file.git/.git refs/heads/master
+'
+
+test_expect_success SMALL_SVN 'fetch each rev of SMALL separately' '
+ reinit_git &&
+ url=$(svnurl small.svn) &&
+
+ for x in `seq 1 1 10`; do
+ git remote add svn_$x "$url.r0-$x"
+ done &&
+ git remote update &&
+
+ git remote add svn "$url" &&
+ git fetch svn &&
+
+ git rev-parse -s remotes/svn_7/master~5 >ref7_2 &&
+ git rev-parse -s remotes/svn/master~8 >ref_2 &&
+ test_cmp ref7_2 ref_2
+'
+
+test_expect_success SMALL_SVN 'fetch updates from SMALL' '
+ reinit_git &&
+ url=$(svnurl link.svn) &&
+ git remote add svn "$url" &&
+
+ ln -sfn small.svn.r0-5 link.svn &&
+ git fetch svn &&
+ test_nr_revs 5 .git refs/remotes/svn/master &&
+
+ ln -sfn small.svn.r0-10 link.svn &&
+ git fetch svn &&
+ test_nr_revs 10 .git refs/remotes/svn/master &&
+
+ git fetch svn &&
+ test_nr_revs 10 .git refs/remotes/svn/master
+'
+
+test_expect_success TINY_SVN 'fetch TINY does not write to refs/heads/master' '
+ reinit_git &&
+ url=$(svnurl tiny.svn) &&
+ git remote add svn "$url" &&
+ git fetch svn &&
+ git show-ref --verify refs/remotes/svn/master &&
+ test_must_fail git show-ref --verify refs/heads/master
+'
+
+test_expect_success SMALL_SVN 'fetch SMALL writes revnum notes' '
+ reinit_git &&
+ url=$(svnurl small.svn) &&
+ git remote add svn "$url" &&
+ git fetch svn &&
+ git log --show-notes=refs/svn-alpha/svn/SVNR --format=%N -1 refs/remotes/svn/master^ >actual.note &&
+ echo r9 >expect.note &&
+ echo >>expect.note &&
+ test_cmp expect.note actual.note
+'
+
+test_expect_success SMALL_SVN 'marks from notes regeneration' '
+ reinit_git &&
+ url=$(svnurl link.svn) &&
+ git remote add svn "$url" &&
+
+ ln -sfn small.svn.r0-5 link.svn &&
+ git fetch svn &&
+ test_nr_revs 5 .git refs/remotes/svn/master &&
+
+ rm -rf .git/info/fast-import/svn-alpha &&
+
+ ln -sfn small.svn.r0-10 link.svn &&
+ git fetch svn &&
+ test_nr_revs 10 .git refs/remotes/svn/master
+'
+
+test_expect_success SMALL_SVN 'clone to bootstrap' '
+ deinit_git &&
+ url=$(svnurl link.svn) &&
+ ln -sfn small.svn.r0-5 link.svn &&
+ git clone -b master "$url" master.git &&
+ {
+ cd master.git &&
+ git branch pub_head refs/svn-alpha/origin/SVNHEAD &&
+ git branch pub_notes refs/svn-alpha/origin/SVNR &&
+ cd ..
+ } &&
+ ln -sfn small.svn.r0-8 link.svn &&
+ git clone master.git slave.git &&
+ {
+ cd slave.git &&
+ git remote add svn "$url" &&
+ git update-ref refs/svn-alpha/svn/SVNHEAD refs/remotes/origin/pub_head &&
+ git update-ref refs/svn-alpha/svn/SVNR refs/remotes/origin/pub_notes &&
+ git fetch svn &&
+ git merge-base refs/svn-alpha/svn/SVNR refs/remotes/origin/pub_notes &&
+ cd ..
+ } &&
+ test_nr_revs 8 slave.git/.git refs/remotes/svn/master
+'
+
+test_expect_success SMALL_SVN 'clone and exchange' '
+ deinit_git &&
+ url=$(svnurl link.svn) &&
+ ln -sfn small.svn.r0-1 link.svn &&
+ git clone -b master "$url" A.git &&
+ git clone -b master "$url" B.git &&
+ test_nr_revs 1 A.git/.git origin/master &&
+ test_nr_revs 1 B.git/.git origin/master &&
+
+ git --git-dir=A.git/.git remote add B B.git &&
+ git --git-dir=B.git/.git remote add A A.git &&
+ refspecs="+refs/svn-alpha/origin/SVNR:refs/svn-alpha/origin/SVNR"
+ refspecs="$refspecs refs/svn-alpha/origin/SVNHEAD:refs/svn-alpha/origin/SVNHEAD"
+
+ ln -sfn small.svn.r0-2 link.svn &&
+ git --git-dir=A.git/.git fetch origin &&
+
+ git --git-dir=B.git/.git fetch -f A $refspecs &&
+
+ ln -sfn small.svn.r0-3 link.svn &&
+ git --git-dir=B.git/.git fetch origin &&
+
+ git --git-dir=A.git/.git fetch B $refspecs &&
+
+ git --git-dir=A.git/.git fetch origin &&
+ git --git-dir=B.git/.git fetch origin &&
+
+ test_nr_revs 3 A.git/.git origin/master &&
+ test_nr_revs 3 B.git/.git origin/master
+'
+
+test_done
^ permalink raw reply related [flat|nested] 10+ messages in thread
* [PATCH v2 1/9] svn-fe: use svnrdump --quiet in remote-svn-alpha
2011-07-13 15:26 [GSoC'11] [PATCH/RFC v2 0/9] remote-svn-alpha updates Dmitry Ivankov
@ 2011-07-13 15:26 ` Dmitry Ivankov
2011-07-13 15:26 ` [PATCH v2 2/9] svn-fe: avoid error on no-op imports " Dmitry Ivankov
` (7 subsequent siblings)
8 siblings, 0 replies; 10+ messages in thread
From: Dmitry Ivankov @ 2011-07-13 15:26 UTC (permalink / raw)
To: git; +Cc: Jonathan Nieder, David Barr, Ramkumar Ramachandra, Dmitry Ivankov
svnrdump by default shows "Dumped revision #n" lines on stderr.
svn-fe does it too on each imported revision, so pass --quiet to
svnrdump. Once process indication is really needed it will be
custom formatted in either svn-fe or remote-svn-alpha.
Signed-off-by: Dmitry Ivankov <divanorama@gmail.com>
---
contrib/svn-fe/git-remote-svn-alpha | 2 +-
1 files changed, 1 insertions(+), 1 deletions(-)
diff --git a/contrib/svn-fe/git-remote-svn-alpha b/contrib/svn-fe/git-remote-svn-alpha
index b2cca9f..61c9b07 100755
--- a/contrib/svn-fe/git-remote-svn-alpha
+++ b/contrib/svn-fe/git-remote-svn-alpha
@@ -15,7 +15,7 @@ usage () {
do_import () {
revs=$1 url=$2
(svnrdump dump --non-interactive --username=Guest --password= \
- -r"$revs" "$url" | svn-fe) 3<&0 || die "FAILURE"
+ -r"$revs" "$url" --quiet | svn-fe) 3<&0 || die "FAILURE"
exec 1>&-
}
--
1.7.3.4
^ permalink raw reply related [flat|nested] 10+ messages in thread
* [PATCH v2 2/9] svn-fe: avoid error on no-op imports in remote-svn-alpha
2011-07-13 15:26 [GSoC'11] [PATCH/RFC v2 0/9] remote-svn-alpha updates Dmitry Ivankov
2011-07-13 15:26 ` [PATCH v2 1/9] svn-fe: use svnrdump --quiet in remote-svn-alpha Dmitry Ivankov
@ 2011-07-13 15:26 ` Dmitry Ivankov
2011-07-13 15:26 ` [PATCH v2 3/9] svn-fe: allow svnadmin instead of svnrdump " Dmitry Ivankov
` (6 subsequent siblings)
8 siblings, 0 replies; 10+ messages in thread
From: Dmitry Ivankov @ 2011-07-13 15:26 UTC (permalink / raw)
To: git; +Cc: Jonathan Nieder, David Barr, Ramkumar Ramachandra, Dmitry Ivankov
Currently helpers' exit codes are not checked by transport_helper.
But a proper helper should check for all possible internal errors
anyway. svnrdump dump --incremental considers -rX:HEAD range as bad
if X is greater than the actual HEAD revision. And there is no option
to change this behavior.
One way to address this issue is to fire one more svn command to get
the HEAD value, but it is one more connection, one more tool (ok,
we can svnrdump HEAD but it looks slow) and possibly one more
password prompt, and maybe even a race condition (if we talk to a
svn servers farm frontend for example).
Another one is to patch svnrdump to report the revision, because
internally it asks svn for it anyway before doing a dump. Longer
term it looks nice, moreover it can be used to display percentage
progress.
Add a wrapper around svnrdump that captures stderr and exit code.
If stderr matches a hardcoded "LOWER cannot be greater than UPPER."
and the exit code is non-zero, don't produce any dump and emulate
exit 0, otherwise the wrapper is transparent. The only side effect
is dup2-ing stdout to fd=6, ugly but fine as only standard
descriptors are used currently.
Signed-off-by: Dmitry Ivankov <divanorama@gmail.com>
---
contrib/svn-fe/git-remote-svn-alpha | 19 +++++++++++++++++--
1 files changed, 17 insertions(+), 2 deletions(-)
diff --git a/contrib/svn-fe/git-remote-svn-alpha b/contrib/svn-fe/git-remote-svn-alpha
index 61c9b07..2eac7d4 100755
--- a/contrib/svn-fe/git-remote-svn-alpha
+++ b/contrib/svn-fe/git-remote-svn-alpha
@@ -12,10 +12,25 @@ usage () {
exit 129
}
+svnrdump_wrap () {
+ exec 6<&1 &&
+
+ EX=$(
+ (svnrdump dump --non-interactive --username=Guest --password= \
+ --quiet "$1" "$2" >&6) 2>&1; test $? -ne 0 || echo Success
+ ) &&
+ if test "z$EX" != "zSuccess"; then
+ if test "z$EX" = "zLOWER cannot be greater than UPPER."; then
+ return 0
+ fi
+ echo "$EX" >&2
+ return 1
+ fi
+}
+
do_import () {
revs=$1 url=$2
- (svnrdump dump --non-interactive --username=Guest --password= \
- -r"$revs" "$url" --quiet | svn-fe) 3<&0 || die "FAILURE"
+ (svnrdump_wrap "$url" -r"$revs" | svn-fe) 3<&0 || die "FAILURE"
exec 1>&-
}
--
1.7.3.4
^ permalink raw reply related [flat|nested] 10+ messages in thread
* [PATCH v2 3/9] svn-fe: allow svnadmin instead of svnrdump in remote-svn-alpha
2011-07-13 15:26 [GSoC'11] [PATCH/RFC v2 0/9] remote-svn-alpha updates Dmitry Ivankov
2011-07-13 15:26 ` [PATCH v2 1/9] svn-fe: use svnrdump --quiet in remote-svn-alpha Dmitry Ivankov
2011-07-13 15:26 ` [PATCH v2 2/9] svn-fe: avoid error on no-op imports " Dmitry Ivankov
@ 2011-07-13 15:26 ` Dmitry Ivankov
2011-07-13 15:26 ` [PATCH v2 4/9] svn-fe: add a test for remote-svn-alpha Dmitry Ivankov
` (5 subsequent siblings)
8 siblings, 0 replies; 10+ messages in thread
From: Dmitry Ivankov @ 2011-07-13 15:26 UTC (permalink / raw)
To: git; +Cc: Jonathan Nieder, David Barr, Ramkumar Ramachandra, Dmitry Ivankov
svnrdump dump and svnadmin dump share a dump format. Their dump
streams representation may be different but they should have the
same effect on a dump consumer. One more difference is svnrdump being
able to dump via any svn remote protocol while svnadmin needs access
to the repository filesystem and also can't produce subdirectory dumps.
But svnrdump is a newer tool and may be unavailable on some systems.
Try to use svnadmin dump for file:// repository urls if there is no
svnrdump in the PATH. First of all this is to be used in tests, where
the repository is indeed local most of the time. Also require svnlook
utility as a lightweight option to get the youngest repository revision.
Signed-off-by: Dmitry Ivankov <divanorama@gmail.com>
---
contrib/svn-fe/git-remote-svn-alpha | 31 ++++++++++++++++++++++++++++++-
1 files changed, 30 insertions(+), 1 deletions(-)
diff --git a/contrib/svn-fe/git-remote-svn-alpha b/contrib/svn-fe/git-remote-svn-alpha
index 2eac7d4..ca9d431 100755
--- a/contrib/svn-fe/git-remote-svn-alpha
+++ b/contrib/svn-fe/git-remote-svn-alpha
@@ -28,9 +28,38 @@ svnrdump_wrap () {
fi
}
+try_svnrdump () {
+ command -v svnrdump >/dev/null &&
+ echo svnrdump_wrap ||
+ true
+}
+
+svnadmin_wrap () {
+ path=${1##file://} &&
+ test "z$path" != "z$1" &&
+ latest=$(svnlook youngest "$path") &&
+ future=$(( $latest + 1 )) &&
+ asked=${2#-r} &&
+ asked=${asked%:*} &&
+ test "$asked" -eq "$future" ||
+ svnadmin dump --incremental --deltas --quiet "$path" "$2"
+}
+
+try_svnadmin () {
+ command -v svnadmin >/dev/null &&
+ command -v svnlook >/dev/null &&
+ echo svnadmin_wrap ||
+ true
+}
+
+SVNDUMP=""
+SVNDUMP=${SVNDUMP:-`try_svnrdump`}
+SVNDUMP=${SVNDUMP:-`try_svnadmin`}
+test -n "$SVNDUMP" || die "neither svnrdump nor svnadmin & svnlook was found"
+
do_import () {
revs=$1 url=$2
- (svnrdump_wrap "$url" -r"$revs" | svn-fe) 3<&0 || die "FAILURE"
+ (eval "$SVNDUMP \"$url\" -r\"$revs\"" | svn-fe) 3<&0 || die "FAILURE"
exec 1>&-
}
--
1.7.3.4
^ permalink raw reply related [flat|nested] 10+ messages in thread
* [PATCH v2 4/9] svn-fe: add a test for remote-svn-alpha
2011-07-13 15:26 [GSoC'11] [PATCH/RFC v2 0/9] remote-svn-alpha updates Dmitry Ivankov
` (2 preceding siblings ...)
2011-07-13 15:26 ` [PATCH v2 3/9] svn-fe: allow svnadmin instead of svnrdump " Dmitry Ivankov
@ 2011-07-13 15:26 ` Dmitry Ivankov
2011-07-13 15:26 ` [PATCH v2 5/9] svn-fe: use svn-fe --no-progress in remote-svn-alpha Dmitry Ivankov
` (4 subsequent siblings)
8 siblings, 0 replies; 10+ messages in thread
From: Dmitry Ivankov @ 2011-07-13 15:26 UTC (permalink / raw)
To: git; +Cc: Jonathan Nieder, David Barr, Ramkumar Ramachandra, Dmitry Ivankov
The test creates a few svn repositories from fixed dumps using svnadmin.
Currently it checks for crashes on first time import mostly.
Signed-off-by: Dmitry Ivankov <divanorama@gmail.com>
---
contrib/svn-fe/t/.gitignore | 3 +
contrib/svn-fe/t/t9010-remote-svn-alpha.sh | 259 ++++++++++++++++++++++++++++
2 files changed, 262 insertions(+), 0 deletions(-)
create mode 100644 contrib/svn-fe/t/.gitignore
create mode 100755 contrib/svn-fe/t/t9010-remote-svn-alpha.sh
diff --git a/contrib/svn-fe/t/.gitignore b/contrib/svn-fe/t/.gitignore
new file mode 100644
index 0000000..4e731dc
--- /dev/null
+++ b/contrib/svn-fe/t/.gitignore
@@ -0,0 +1,3 @@
+/trash directory*
+/test-results
+/.prove
diff --git a/contrib/svn-fe/t/t9010-remote-svn-alpha.sh b/contrib/svn-fe/t/t9010-remote-svn-alpha.sh
new file mode 100755
index 0000000..92bc4b2
--- /dev/null
+++ b/contrib/svn-fe/t/t9010-remote-svn-alpha.sh
@@ -0,0 +1,259 @@
+#!/bin/sh
+
+test_description='check svn-alpha remote helper'
+
+PATH=$(pwd)/..:$PATH
+TEST_DIRECTORY=$(pwd)/../../../t
+. $TEST_DIRECTORY/test-lib.sh
+
+if command -v svnrdump >/dev/null; then
+ test_set_prereq SVNRDUMP
+fi
+
+deinit_git () {
+ rm -fr .git
+}
+
+reinit_git () {
+ deinit_git &&
+ git init
+}
+
+properties () {
+ while test "$#" -ne 0
+ do
+ property="$1" &&
+ value="$2" &&
+ printf "%s\n" "K ${#property}" &&
+ printf "%s\n" "$property" &&
+ printf "%s\n" "V ${#value}" &&
+ printf "%s\n" "$value" &&
+ shift 2 ||
+ return 1
+ done
+}
+
+text_no_props () {
+ text="$1
+" &&
+ printf "%s\n" "Prop-content-length: 10" &&
+ printf "%s\n" "Text-content-length: ${#text}" &&
+ printf "%s\n" "Content-length: $((${#text} + 10))" &&
+ printf "%s\n" "" "PROPS-END" &&
+ printf "%s\n" "$text"
+}
+
+dump_to_svnrepo () {
+ dump="$1" &&
+ path="$2" &&
+ svnadmin create "$path" &&
+ svnadmin load "$path" < "$dump"
+}
+
+svnurl () {
+ printf "svn-alpha::file://%s/%s" "$(pwd)" "$1"
+}
+
+test_nr_revs () {
+ n=$1 &&
+ repo=$2 &&
+ ref=$3 &&
+ git --git-dir="$repo" log --format=oneline "$ref" >revs &&
+ test_line_count = "$n" revs
+}
+
+test_expect_success 'svnadmin is present' '
+ command -v svnadmin &&
+ test_set_prereq SVNADMIN
+'
+
+test_expect_success SVNADMIN 'create empty svnrepo' '
+ echo "SVN-fs-dump-format-version: 2" > empty.dump &&
+ dump_to_svnrepo empty.dump empty.svn &&
+ test_set_prereq EMPTY_SVN
+'
+
+test_expect_success SVNADMIN 'create tiny svnrepo' '
+ {
+ properties \
+ svn:author author@example.com \
+ svn:date "1999-02-01T00:01:002.000000Z" \
+ svn:log "add directory with some files in it" &&
+ echo PROPS-END
+ } >props &&
+ {
+ cat <<-EOF &&
+ SVN-fs-dump-format-version: 3
+
+ Revision-number: 1
+ EOF
+ echo Prop-content-length: $(wc -c <props) &&
+ echo Content-length: $(wc -c <props) &&
+ echo &&
+ cat props &&
+ cat <<-\EOF &&
+
+ Node-path: directory
+ Node-kind: dir
+ Node-action: add
+ Prop-content-length: 10
+ Content-length: 10
+
+ PROPS-END
+ Node-path: directory/somefile
+ Node-kind: file
+ Node-action: add
+ EOF
+ text_no_props hi
+ } >tiny.dump &&
+ dump_to_svnrepo tiny.dump tiny.svn &&
+ test_set_prereq TINY_SVN
+'
+
+test_expect_success SVNADMIN 'create small svndump' '
+ {
+ properties \
+ svn:author author@example.com \
+ svn:date "1999-02-01T00:01:002.000000Z" \
+ svn:log "add directory with some files in it" &&
+ echo PROPS-END
+ } >props &&
+ {
+ echo Prop-content-length: $(wc -c <props) &&
+ echo Content-length: $(wc -c <props) &&
+ echo &&
+ cat props
+ } >props_dump &&
+ cat >small.dump.r0 <<-EOF &&
+ SVN-fs-dump-format-version: 3
+ EOF
+ for x in `seq 1 1 10`; do
+ {
+ echo &&
+ echo "Revision-number: $x" &&
+ cat props_dump &&
+
+ if test "$x" -eq "1"; then
+ cat <<-\EOF
+
+ Node-path: directory
+ Node-kind: dir
+ Node-action: add
+ Prop-content-length: 10
+ Content-length: 10
+
+ PROPS-END
+
+ EOF
+ fi
+ echo "Node-path: directory/somefile$x" &&
+ echo "Node-kind: file" &&
+ echo "Node-action: add" &&
+ text_no_props hi
+ } >small.dump.r$x
+ done &&
+ test_set_prereq SMALL_SVNDUMP
+'
+
+test_expect_success SMALL_SVNDUMP 'create small svnrepo' '
+ rm -rf small.svn* &&
+ svnadmin create small.svn &&
+ for x in `seq 1 1 10`; do
+ cat small.dump.r0 small.dump.r$x >part.dump &&
+ svnadmin load small.svn <part.dump &&
+ cp -r small.svn small.svn.r0-$x
+ done &&
+ test_set_prereq SMALL_SVN
+'
+
+test_expect_failure EMPTY_SVN 'fetch empty' '
+ reinit_git &&
+ url=$(svnurl empty.svn) &&
+ git remote add svn "$url" &&
+ git fetch svn
+'
+
+test_expect_failure TINY_SVN 'clone tiny' '
+ deinit_git &&
+ url=$(svnurl tiny.svn) &&
+ git clone "$url" tiny1.git &&
+ test_nr_revs 1 tiny1.git refs/remotes/origin/master
+'
+
+test_expect_success TINY_SVN 'clone --mirror tiny' '
+ deinit_git &&
+ url=$(svnurl tiny.svn) &&
+ git clone --mirror "$url" tiny2.git &&
+ test_nr_revs 1 tiny2.git refs/heads/master
+'
+
+test_expect_success TINY_SVN 'clone --bare tiny' '
+ deinit_git &&
+ url=$(svnurl tiny.svn) &&
+ git clone --bare "$url" tiny3.git &&
+ test_nr_revs 1 tiny3.git refs/heads/master
+'
+
+test_expect_success TINY_SVN 'clone -b master tiny' '
+ deinit_git &&
+ url=$(svnurl tiny.svn) &&
+ git clone -b master "$url" tiny4.git &&
+ test_nr_revs 1 tiny4.git/.git refs/heads/master
+'
+
+test_expect_success SMALL_SVN 'clone -b master small' '
+ deinit_git &&
+ url=$(svnurl small.svn) &&
+ git clone -b master "$url" small.git &&
+ test_nr_revs 10 small.git/.git refs/heads/master
+'
+
+test_expect_success TINY_SVN,SVNRDUMP 'no crash on clone url/path' '
+ deinit_git &&
+ url=$(svnurl small.svn)/directory &&
+ git clone -b master "$url" small_dir.git &&
+ test_nr_revs 10 small_dir.git/.git refs/heads/master
+'
+
+test_expect_success TINY_SVN,SVNRDUMP 'no crash on clone url/path/file' '
+ deinit_git &&
+ url=$(svnurl small.svn)/directory/somefile3 &&
+ git clone -b master "$url" small_dir_file.git &&
+ test_nr_revs 10 small_dir_file.git/.git refs/heads/master
+'
+
+test_expect_success SMALL_SVN 'fetch each rev of SMALL separately' '
+ reinit_git &&
+ url=$(svnurl small.svn) &&
+
+ for x in `seq 1 1 10`; do
+ git remote add svn_$x "$url.r0-$x"
+ done &&
+ git remote update &&
+
+ git remote add svn "$url" &&
+ git fetch svn &&
+
+ git rev-parse -s remotes/svn_7/master~5 >ref7_2 &&
+ git rev-parse -s remotes/svn/master~8 >ref_2 &&
+ test_cmp ref7_2 ref_2
+'
+
+test_expect_success SMALL_SVN 'fetch updates from SMALL' '
+ reinit_git &&
+ url=$(svnurl link.svn) &&
+ git remote add svn "$url" &&
+
+ ln -sfn small.svn.r0-5 link.svn &&
+ git fetch svn &&
+ test_nr_revs 5 .git refs/remotes/svn/master &&
+
+ ln -sfn small.svn.r0-10 link.svn &&
+ git fetch svn &&
+ test_nr_revs 10 .git refs/remotes/svn/master &&
+
+ git fetch svn &&
+ test_nr_revs 10 .git refs/remotes/svn/master
+'
+
+test_done
--
1.7.3.4
^ permalink raw reply related [flat|nested] 10+ messages in thread
* [PATCH v2 5/9] svn-fe: use svn-fe --no-progress in remote-svn-alpha
2011-07-13 15:26 [GSoC'11] [PATCH/RFC v2 0/9] remote-svn-alpha updates Dmitry Ivankov
` (3 preceding siblings ...)
2011-07-13 15:26 ` [PATCH v2 4/9] svn-fe: add a test for remote-svn-alpha Dmitry Ivankov
@ 2011-07-13 15:26 ` Dmitry Ivankov
2011-07-13 15:26 ` [PATCH v2 6/9] svn-fe: use proper refspec " Dmitry Ivankov
` (3 subsequent siblings)
8 siblings, 0 replies; 10+ messages in thread
From: Dmitry Ivankov @ 2011-07-13 15:26 UTC (permalink / raw)
To: git; +Cc: Jonathan Nieder, David Barr, Ramkumar Ramachandra, Dmitry Ivankov
svn-fe by default produces a progress line for each imported revision.
For large repos it stresses the terminal and a user, possibly scrolls
away error messages.
Just disable progress lines for now.
Signed-off-by: Dmitry Ivankov <divanorama@gmail.com>
---
contrib/svn-fe/git-remote-svn-alpha | 2 +-
1 files changed, 1 insertions(+), 1 deletions(-)
diff --git a/contrib/svn-fe/git-remote-svn-alpha b/contrib/svn-fe/git-remote-svn-alpha
index ca9d431..ed8da7b 100755
--- a/contrib/svn-fe/git-remote-svn-alpha
+++ b/contrib/svn-fe/git-remote-svn-alpha
@@ -59,7 +59,7 @@ test -n "$SVNDUMP" || die "neither svnrdump nor svnadmin & svnlook was found"
do_import () {
revs=$1 url=$2
- (eval "$SVNDUMP \"$url\" -r\"$revs\"" | svn-fe) 3<&0 || die "FAILURE"
+ (eval "$SVNDUMP \"$url\" -r\"$revs\"" | svn-fe --no-progress) 3<&0 || die "FAILURE"
exec 1>&-
}
--
1.7.3.4
^ permalink raw reply related [flat|nested] 10+ messages in thread
* [PATCH v2 6/9] svn-fe: use proper refspec in remote-svn-alpha
2011-07-13 15:26 [GSoC'11] [PATCH/RFC v2 0/9] remote-svn-alpha updates Dmitry Ivankov
` (4 preceding siblings ...)
2011-07-13 15:26 ` [PATCH v2 5/9] svn-fe: use svn-fe --no-progress in remote-svn-alpha Dmitry Ivankov
@ 2011-07-13 15:26 ` Dmitry Ivankov
2011-07-13 15:26 ` [PATCH v2 7/9] svn-fe: write svnrev notes " Dmitry Ivankov
` (2 subsequent siblings)
8 siblings, 0 replies; 10+ messages in thread
From: Dmitry Ivankov @ 2011-07-13 15:26 UTC (permalink / raw)
To: git; +Cc: Jonathan Nieder, David Barr, Ramkumar Ramachandra, Dmitry Ivankov
Create a per repository name private refs namespace as suggested by
the remote helpers documentation. This allows to have several svn
remotes at the same time. And also this way we don't overwrite the
local branch master.
The refpec is:
HEAD:refs/svn-alpha/$reponame/SVNHEAD
refs/heads/master:refs/svn-alpha/$reponame/SVNHEAD
Signed-off-by: Dmitry Ivankov <divanorama@gmail.com>
---
contrib/svn-fe/git-remote-svn-alpha | 18 ++++++++++--------
contrib/svn-fe/t/t9010-remote-svn-alpha.sh | 9 +++++++++
2 files changed, 19 insertions(+), 8 deletions(-)
diff --git a/contrib/svn-fe/git-remote-svn-alpha b/contrib/svn-fe/git-remote-svn-alpha
index ed8da7b..b410302 100755
--- a/contrib/svn-fe/git-remote-svn-alpha
+++ b/contrib/svn-fe/git-remote-svn-alpha
@@ -58,8 +58,8 @@ SVNDUMP=${SVNDUMP:-`try_svnadmin`}
test -n "$SVNDUMP" || die "neither svnrdump nor svnadmin & svnlook was found"
do_import () {
- revs=$1 url=$2
- (eval "$SVNDUMP \"$url\" -r\"$revs\"" | svn-fe --no-progress) 3<&0 || die "FAILURE"
+ revs=$1 url=$2 dst=$3
+ (eval "$SVNDUMP \"$url\" -r\"$revs\"" | svn-fe --ref="$dst" --no-progress) 3<&0 || die "FAILURE"
exec 1>&-
}
@@ -68,29 +68,31 @@ usage 'git remote-svn-alpha <repository> <URL> < commandlist'
repo=$1
url=$2
need_import=""
+remote_ref="refs/heads/master"
+private_ref="refs/svn-alpha/$repo/SVNHEAD"
while read -r cmd args
do
case $cmd in
capabilities)
echo import
- echo "refspec HEAD:refs/heads/master"
- echo "refspec refs/heads/master:refs/heads/master"
+ echo "refspec HEAD:$private_ref"
+ echo "refspec $remote_ref:$private_ref"
echo
;;
list)
- echo '? HEAD'
- echo '? refs/heads/master'
+ echo "? HEAD"
+ echo "? $remote_ref"
echo
;;
import)
- test "$args" = "HEAD" || test "$args" = "refs/heads/master" ||
+ test "$args" = "HEAD" || test "$args" = "$remote_ref" ||
die "remote-svn-alpha: unsupported import ref argument: $args"
need_import="yes"
;;
'')
test "$need_import" = "yes" || exit 0
- do_import 0:HEAD "$url"
+ do_import 0:HEAD "$url" "$private_ref"
need_import=""
;;
*)
diff --git a/contrib/svn-fe/t/t9010-remote-svn-alpha.sh b/contrib/svn-fe/t/t9010-remote-svn-alpha.sh
index 92bc4b2..b0f41cb 100755
--- a/contrib/svn-fe/t/t9010-remote-svn-alpha.sh
+++ b/contrib/svn-fe/t/t9010-remote-svn-alpha.sh
@@ -256,4 +256,13 @@ test_expect_success SMALL_SVN 'fetch updates from SMALL' '
test_nr_revs 10 .git refs/remotes/svn/master
'
+test_expect_success TINY_SVN 'fetch TINY does not write to refs/heads/master' '
+ reinit_git &&
+ url=$(svnurl tiny.svn) &&
+ git remote add svn "$url" &&
+ git fetch svn &&
+ git show-ref --verify refs/remotes/svn/master &&
+ test_must_fail git show-ref --verify refs/heads/master
+'
+
test_done
--
1.7.3.4
^ permalink raw reply related [flat|nested] 10+ messages in thread
* [PATCH v2 7/9] svn-fe: write svnrev notes in remote-svn-alpha
2011-07-13 15:26 [GSoC'11] [PATCH/RFC v2 0/9] remote-svn-alpha updates Dmitry Ivankov
` (5 preceding siblings ...)
2011-07-13 15:26 ` [PATCH v2 6/9] svn-fe: use proper refspec " Dmitry Ivankov
@ 2011-07-13 15:26 ` Dmitry Ivankov
2011-07-13 15:26 ` [PATCH v2 8/9] svn-fe: import incrementally in svn-remote-alpha Dmitry Ivankov
2011-07-13 15:26 ` [PATCH v2 9/9] svn-fe: reuse import-marks in remote-svn-alpha Dmitry Ivankov
8 siblings, 0 replies; 10+ messages in thread
From: Dmitry Ivankov @ 2011-07-13 15:26 UTC (permalink / raw)
To: git; +Cc: Jonathan Nieder, David Barr, Ramkumar Ramachandra, Dmitry Ivankov
For each imported svn commit write "rN" note in a private namespace.
These can be viewed with somewhat ugly
$ git log --show-notes=refs/svn-alpha/<reponame>/SVNR
But also these notes can be used to import svn history incrementally.
Add a simple test for these notes.
Signed-off-by: Dmitry Ivankov <divanorama@gmail.com>
---
contrib/svn-fe/git-remote-svn-alpha | 9 ++++++---
contrib/svn-fe/t/t9010-remote-svn-alpha.sh | 11 +++++++++++
2 files changed, 17 insertions(+), 3 deletions(-)
diff --git a/contrib/svn-fe/git-remote-svn-alpha b/contrib/svn-fe/git-remote-svn-alpha
index b410302..c2af393 100755
--- a/contrib/svn-fe/git-remote-svn-alpha
+++ b/contrib/svn-fe/git-remote-svn-alpha
@@ -58,8 +58,8 @@ SVNDUMP=${SVNDUMP:-`try_svnadmin`}
test -n "$SVNDUMP" || die "neither svnrdump nor svnadmin & svnlook was found"
do_import () {
- revs=$1 url=$2 dst=$3
- (eval "$SVNDUMP \"$url\" -r\"$revs\"" | svn-fe --ref="$dst" --no-progress) 3<&0 || die "FAILURE"
+ revs=$1 url=$2 dst=$3 notes=$4
+ (eval "$SVNDUMP \"$url\" -r\"$revs\"" | svn-fe --ref="$dst" --notes_ref="$notes" --no-progress) 3<&0 || die "FAILURE"
exec 1>&-
}
@@ -70,6 +70,8 @@ url=$2
need_import=""
remote_ref="refs/heads/master"
private_ref="refs/svn-alpha/$repo/SVNHEAD"
+remote_notes="refs/notes/svnr"
+private_notes="refs/svn-alpha/$repo/SVNR"
while read -r cmd args
do
@@ -78,6 +80,7 @@ do
echo import
echo "refspec HEAD:$private_ref"
echo "refspec $remote_ref:$private_ref"
+ echo "refspec $remote_notes:$private_notes"
echo
;;
list)
@@ -92,7 +95,7 @@ do
;;
'')
test "$need_import" = "yes" || exit 0
- do_import 0:HEAD "$url" "$private_ref"
+ do_import 0:HEAD "$url" "$private_ref" "$private_notes"
need_import=""
;;
*)
diff --git a/contrib/svn-fe/t/t9010-remote-svn-alpha.sh b/contrib/svn-fe/t/t9010-remote-svn-alpha.sh
index b0f41cb..5102428 100755
--- a/contrib/svn-fe/t/t9010-remote-svn-alpha.sh
+++ b/contrib/svn-fe/t/t9010-remote-svn-alpha.sh
@@ -265,4 +265,15 @@ test_expect_success TINY_SVN 'fetch TINY does not write to refs/heads/master' '
test_must_fail git show-ref --verify refs/heads/master
'
+test_expect_success SMALL_SVN 'fetch SMALL writes revnum notes' '
+ reinit_git &&
+ url=$(svnurl small.svn) &&
+ git remote add svn "$url" &&
+ git fetch svn &&
+ git log --show-notes=refs/svn-alpha/svn/SVNR --format=%N -1 refs/remotes/svn/master^ >actual.note &&
+ echo r9 >expect.note &&
+ echo >>expect.note &&
+ test_cmp expect.note actual.note
+'
+
test_done
--
1.7.3.4
^ permalink raw reply related [flat|nested] 10+ messages in thread
* [PATCH v2 8/9] svn-fe: import incrementally in svn-remote-alpha
2011-07-13 15:26 [GSoC'11] [PATCH/RFC v2 0/9] remote-svn-alpha updates Dmitry Ivankov
` (6 preceding siblings ...)
2011-07-13 15:26 ` [PATCH v2 7/9] svn-fe: write svnrev notes " Dmitry Ivankov
@ 2011-07-13 15:26 ` Dmitry Ivankov
2011-07-13 15:26 ` [PATCH v2 9/9] svn-fe: reuse import-marks in remote-svn-alpha Dmitry Ivankov
8 siblings, 0 replies; 10+ messages in thread
From: Dmitry Ivankov @ 2011-07-13 15:26 UTC (permalink / raw)
To: git; +Cc: Jonathan Nieder, David Barr, Ramkumar Ramachandra, Dmitry Ivankov
svn-remote-alpha already maintains commits tree imported from svn
and a mapping to the original svn revision numbers as notes content.
svn-fe --incremental requires import marks from previous revisions.
Just generate import marks from notes on each import. This is cheap
on small repositories, fine for a few tests.
Signed-off-by: Dmitry Ivankov <divanorama@gmail.com>
---
contrib/svn-fe/git-remote-svn-alpha | 57 +++++++++++++++++++++-
contrib/svn-fe/t/t9010-remote-svn-alpha.sh | 72 ++++++++++++++++++++++++++++
2 files changed, 128 insertions(+), 1 deletions(-)
diff --git a/contrib/svn-fe/git-remote-svn-alpha b/contrib/svn-fe/git-remote-svn-alpha
index c2af393..f4e737d 100755
--- a/contrib/svn-fe/git-remote-svn-alpha
+++ b/contrib/svn-fe/git-remote-svn-alpha
@@ -57,9 +57,60 @@ SVNDUMP=${SVNDUMP:-`try_svnrdump`}
SVNDUMP=${SVNDUMP:-`try_svnadmin`}
test -n "$SVNDUMP" || die "neither svnrdump nor svnadmin & svnlook was found"
+git_dir=""
+marks_dir="info/fast-import/svn-alpha/$repo"
+
+do_gen_marks () {
+ notes=$1
+ test "z$git_dir" != "$z" || die "we were not told the gitdir"
+ git --git-dir="$git_dir" ls-tree -r "$notes" |
+ {
+ while read -r mode type sha path
+ do
+ data=$(git --git-dir="$git_dir" cat-file blob $sha | tail -n 1)
+ data=${data##r}
+ commit=$(echo $path | tr -d /)
+ echo ":$data $commit"
+ done
+ }
+}
+
+prepare_marks () {
+ dst=$1 notes=$2 rev=$3
+ test "z$git_dir" != "$z" || die "we were not told the gitdir"
+ mkdir -p "$git_dir/$marks_dir"
+ if test ! -f "$git_dir/$marks_dir/marks"; then
+ touch "$git_dir/$marks_dir/marks"
+ fi
+ if test "$rev" = "-1"; then
+ return 0
+ fi
+ do_gen_marks "$notes" >"$git_dir/$marks_dir/marks"
+}
+
+last_imported_rev () {
+ dst=$1 notes=$2
+ test "z$git_dir" != "$z" || die "we were not told the gitdir"
+ git --git-dir="$git_dir" show-ref -q --verify $dst || {
+ echo "-1"
+ return 0
+ }
+ rev=$(git --git-dir="$git_dir" log --show-notes="$notes" -1 --format=%N $dst)
+ rev=${rev##r}
+ test "z$rev" != "z" || {
+ die "remote HEAD has no note"
+ }
+ echo $rev
+}
+
do_import () {
revs=$1 url=$2 dst=$3 notes=$4
- (eval "$SVNDUMP \"$url\" -r\"$revs\"" | svn-fe --ref="$dst" --notes_ref="$notes" --no-progress) 3<&0 || die "FAILURE"
+ rev=$(last_imported_rev "$dst" "$notes")
+ start_rev=$(($rev + 1))
+ revs="$start_rev:HEAD"
+ prepare_marks $dst $notes $rev
+ echo "feature import-marks=$git_dir/$marks_dir/marks"
+ (eval "$SVNDUMP \"$url\" -r\"$revs\"" | svn-fe --ref="$dst" --notes-ref="$notes" --incremental --no-progress) 3<&0 || die "FAILURE"
exec 1>&-
}
@@ -78,11 +129,15 @@ do
case $cmd in
capabilities)
echo import
+ echo gitdir
echo "refspec HEAD:$private_ref"
echo "refspec $remote_ref:$private_ref"
echo "refspec $remote_notes:$private_notes"
echo
;;
+ gitdir)
+ git_dir="$args"
+ ;;
list)
echo "? HEAD"
echo "? $remote_ref"
diff --git a/contrib/svn-fe/t/t9010-remote-svn-alpha.sh b/contrib/svn-fe/t/t9010-remote-svn-alpha.sh
index 5102428..9d9aca0 100755
--- a/contrib/svn-fe/t/t9010-remote-svn-alpha.sh
+++ b/contrib/svn-fe/t/t9010-remote-svn-alpha.sh
@@ -276,4 +276,76 @@ test_expect_success SMALL_SVN 'fetch SMALL writes revnum notes' '
test_cmp expect.note actual.note
'
+test_expect_success SMALL_SVN 'marks from notes regeneration' '
+ reinit_git &&
+ url=$(svnurl link.svn) &&
+ git remote add svn "$url" &&
+
+ ln -sfn small.svn.r0-5 link.svn &&
+ git fetch svn &&
+ test_nr_revs 5 .git refs/remotes/svn/master &&
+
+ rm -rf .git/info/fast-import/svn-alpha &&
+
+ ln -sfn small.svn.r0-10 link.svn &&
+ git fetch svn &&
+ test_nr_revs 10 .git refs/remotes/svn/master
+'
+
+test_expect_success SMALL_SVN 'clone to bootstrap' '
+ deinit_git &&
+ url=$(svnurl link.svn) &&
+ ln -sfn small.svn.r0-5 link.svn &&
+ git clone -b master "$url" master.git &&
+ {
+ cd master.git &&
+ git branch pub_head refs/svn-alpha/origin/SVNHEAD &&
+ git branch pub_notes refs/svn-alpha/origin/SVNR &&
+ cd ..
+ } &&
+ ln -sfn small.svn.r0-8 link.svn &&
+ git clone master.git slave.git &&
+ {
+ cd slave.git &&
+ git remote add svn "$url" &&
+ git update-ref refs/svn-alpha/svn/SVNHEAD refs/remotes/origin/pub_head &&
+ git update-ref refs/svn-alpha/svn/SVNR refs/remotes/origin/pub_notes &&
+ git fetch svn &&
+ git merge-base refs/svn-alpha/svn/SVNR refs/remotes/origin/pub_notes &&
+ cd ..
+ } &&
+ test_nr_revs 8 slave.git/.git refs/remotes/svn/master
+'
+
+test_expect_success SMALL_SVN 'clone and exchange' '
+ deinit_git &&
+ url=$(svnurl link.svn) &&
+ ln -sfn small.svn.r0-1 link.svn &&
+ git clone -b master "$url" A.git &&
+ git clone -b master "$url" B.git &&
+ test_nr_revs 1 A.git/.git origin/master &&
+ test_nr_revs 1 B.git/.git origin/master &&
+
+ git --git-dir=A.git/.git remote add B B.git &&
+ git --git-dir=B.git/.git remote add A A.git &&
+ refspecs="+refs/svn-alpha/origin/SVNR:refs/svn-alpha/origin/SVNR"
+ refspecs="$refspecs refs/svn-alpha/origin/SVNHEAD:refs/svn-alpha/origin/SVNHEAD"
+
+ ln -sfn small.svn.r0-2 link.svn &&
+ git --git-dir=A.git/.git fetch origin &&
+
+ git --git-dir=B.git/.git fetch -f A $refspecs &&
+
+ ln -sfn small.svn.r0-3 link.svn &&
+ git --git-dir=B.git/.git fetch origin &&
+
+ git --git-dir=A.git/.git fetch B $refspecs &&
+
+ git --git-dir=A.git/.git fetch origin &&
+ git --git-dir=B.git/.git fetch origin &&
+
+ test_nr_revs 3 A.git/.git origin/master &&
+ test_nr_revs 3 B.git/.git origin/master
+'
+
test_done
--
1.7.3.4
^ permalink raw reply related [flat|nested] 10+ messages in thread
* [PATCH v2 9/9] svn-fe: reuse import-marks in remote-svn-alpha
2011-07-13 15:26 [GSoC'11] [PATCH/RFC v2 0/9] remote-svn-alpha updates Dmitry Ivankov
` (7 preceding siblings ...)
2011-07-13 15:26 ` [PATCH v2 8/9] svn-fe: import incrementally in svn-remote-alpha Dmitry Ivankov
@ 2011-07-13 15:26 ` Dmitry Ivankov
8 siblings, 0 replies; 10+ messages in thread
From: Dmitry Ivankov @ 2011-07-13 15:26 UTC (permalink / raw)
To: git; +Cc: Jonathan Nieder, David Barr, Ramkumar Ramachandra, Dmitry Ivankov
remote-svn-alpha needs to prepare import marks file when doing an
incremental import. This file can be left after previous imports.
Take the last imported commit and a note with corresponding svn
revision number. If the marks file exists and defines a :revision
mark pointing to the last imported commit, consider it valid.
It may accidentally contain newer revisions, they are silently
ignored. In practice the check for :revision last_commit should
be sufficient to rely on this file defining any lesser revision
marks needed correctly.
So if the import marks file looks valid, do not regenerate it from
the notes tree and have save some performance saved.
Signed-off-by: Dmitry Ivankov <divanorama@gmail.com>
---
contrib/svn-fe/git-remote-svn-alpha | 26 ++++++++++++++++++++++----
1 files changed, 22 insertions(+), 4 deletions(-)
diff --git a/contrib/svn-fe/git-remote-svn-alpha b/contrib/svn-fe/git-remote-svn-alpha
index f4e737d..8096ef2 100755
--- a/contrib/svn-fe/git-remote-svn-alpha
+++ b/contrib/svn-fe/git-remote-svn-alpha
@@ -78,14 +78,31 @@ do_gen_marks () {
prepare_marks () {
dst=$1 notes=$2 rev=$3
test "z$git_dir" != "$z" || die "we were not told the gitdir"
- mkdir -p "$git_dir/$marks_dir"
- if test ! -f "$git_dir/$marks_dir/marks"; then
- touch "$git_dir/$marks_dir/marks"
+ path="$git_dir/$marks_dir"
+ mkdir -p "$path"
+ path="$path/marks"
+
+ if test ! -f "$path"; then
+ touch "$path"
fi
if test "$rev" = "-1"; then
return 0
fi
- do_gen_marks "$notes" >"$git_dir/$marks_dir/marks"
+ mark_sha=""
+ {
+ while read -r m sha
+ do
+ if test "$m" = ":$rev"; then
+ mark_sha="$sha"
+ fi
+ done
+ } <"$path"
+ dst_sha=$( git rev-parse "$dst" )
+ if test -n "$mark_sha"; then
+ test "$mark_sha" = "$dst_sha" || die "latest mark and note diverge"
+ else
+ do_gen_marks "$notes" >"$path"
+ fi
}
last_imported_rev () {
@@ -110,6 +127,7 @@ do_import () {
revs="$start_rev:HEAD"
prepare_marks $dst $notes $rev
echo "feature import-marks=$git_dir/$marks_dir/marks"
+ echo "feature export-marks=$git_dir/$marks_dir/marks"
(eval "$SVNDUMP \"$url\" -r\"$revs\"" | svn-fe --ref="$dst" --notes-ref="$notes" --incremental --no-progress) 3<&0 || die "FAILURE"
exec 1>&-
}
--
1.7.3.4
^ permalink raw reply related [flat|nested] 10+ messages in thread
end of thread, other threads:[~2011-07-13 15:27 UTC | newest]
Thread overview: 10+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2011-07-13 15:26 [GSoC'11] [PATCH/RFC v2 0/9] remote-svn-alpha updates Dmitry Ivankov
2011-07-13 15:26 ` [PATCH v2 1/9] svn-fe: use svnrdump --quiet in remote-svn-alpha Dmitry Ivankov
2011-07-13 15:26 ` [PATCH v2 2/9] svn-fe: avoid error on no-op imports " Dmitry Ivankov
2011-07-13 15:26 ` [PATCH v2 3/9] svn-fe: allow svnadmin instead of svnrdump " Dmitry Ivankov
2011-07-13 15:26 ` [PATCH v2 4/9] svn-fe: add a test for remote-svn-alpha Dmitry Ivankov
2011-07-13 15:26 ` [PATCH v2 5/9] svn-fe: use svn-fe --no-progress in remote-svn-alpha Dmitry Ivankov
2011-07-13 15:26 ` [PATCH v2 6/9] svn-fe: use proper refspec " Dmitry Ivankov
2011-07-13 15:26 ` [PATCH v2 7/9] svn-fe: write svnrev notes " Dmitry Ivankov
2011-07-13 15:26 ` [PATCH v2 8/9] svn-fe: import incrementally in svn-remote-alpha Dmitry Ivankov
2011-07-13 15:26 ` [PATCH v2 9/9] svn-fe: reuse import-marks in remote-svn-alpha Dmitry Ivankov
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).