All of lore.kernel.org
 help / color / mirror / Atom feed
From: Jonathan Nieder <jrnieder@gmail.com>
To: git@vger.kernel.org
Cc: Ramkumar Ramachandra <artagnon@gmail.com>,
	Sverre Rabbelier <srabbelier@gmail.com>,
	David Barr <david.barr@cordelta.com>
Subject: [PATCH 2/2] vcs-svn: Allow simple v3 dumps (no deltas yet)
Date: Wed, 17 Nov 2010 23:03:51 -0600	[thread overview]
Message-ID: <20101118050351.GC14861@burratino> (raw)
In-Reply-To: <20101118050023.GA14861@burratino>

From: David Barr <david.barr@cordelta.com>

Since the dumpfile version 1 days, the Subversion dump format
gained some new fields:

 - a unique identifier for the repository (version 2 format)
 - whether the text and properties for a node should be
   interpreted as deltas
 - checksums for a delta's preimage
 - SHA-1 sums as alternatives to the existing MD5 checksums for
   copy source and the payload (delta).

For now what is relevant to us is the Text-delta and Prop-delta
fields, since not noticing these causes a dump file to be
misinterpreted (see the previous commit).

[jn: with tests]

Signed-off-by: David Barr <david.barr@cordelta.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 t/t9010-svn-fe.sh |  350 ++++++++++++++++++++++++++++++++++++++++++++++++++++-
 vcs-svn/svndump.c |   21 +++-
 2 files changed, 365 insertions(+), 6 deletions(-)

diff --git a/t/t9010-svn-fe.sh b/t/t9010-svn-fe.sh
index e9e46ea..be5372a 100755
--- a/t/t9010-svn-fe.sh
+++ b/t/t9010-svn-fe.sh
@@ -9,6 +9,30 @@ reinit_git () {
 	git init
 }
 
+properties () {
+	while test "$#" -ne 0
+	do
+		property="$1" &&
+		value="$2" &&
+		printf "%s\n" "K ${#property}" &&
+		printf "%s\n" "$property" &&
+		printf "%s\n" "V ${#value}" &&
+		printf "%s\n" "$value" &&
+		shift 2 ||
+		return 1
+	done
+}
+
+text_no_props () {
+	text="$1
+" &&
+	printf "%s\n" "Prop-content-length: 10" &&
+	printf "%s\n" "Text-content-length: ${#text}" &&
+	printf "%s\n" "Content-length: $((${#text} + 10))" &&
+	printf "%s\n" "" "PROPS-END" &&
+	printf "%s\n" "$text"
+}
+
 >empty
 
 test_expect_success 'empty dump' '
@@ -18,13 +42,333 @@ test_expect_success 'empty dump' '
 	git fast-import <stream
 '
 
-test_expect_success 'v3 dumps not supported' '
+test_expect_success 'v4 dumps not supported' '
 	reinit_git &&
-	echo "SVN-fs-dump-format-version: 3" >input &&
-	test_must_fail test-svn-fe input >stream &&
+	echo "SVN-fs-dump-format-version: 4" >v4.dump &&
+	test_must_fail test-svn-fe v4.dump >stream &&
 	test_cmp empty stream
 '
 
+test_expect_failure 'empty revision' '
+	reinit_git &&
+	printf "rev <nobody, nobody@local>: %s\n" "" "" >expect &&
+	cat >emptyrev.dump <<-\EOF &&
+	SVN-fs-dump-format-version: 3
+
+	Revision-number: 1
+	Prop-content-length: 0
+	Content-length: 0
+
+	Revision-number: 2
+	Prop-content-length: 0
+	Content-length: 0
+
+	EOF
+	test-svn-fe emptyrev.dump >stream &&
+	git fast-import <stream &&
+	git log -p --format="rev <%an, %ae>: %s" HEAD >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'empty properties' '
+	reinit_git &&
+	printf "rev <nobody, nobody@local>: %s\n" "" "" >expect &&
+	cat >emptyprop.dump <<-\EOF &&
+	SVN-fs-dump-format-version: 3
+
+	Revision-number: 1
+	Prop-content-length: 10
+	Content-length: 10
+
+	PROPS-END
+
+	Revision-number: 2
+	Prop-content-length: 10
+	Content-length: 10
+
+	PROPS-END
+	EOF
+	test-svn-fe emptyprop.dump >stream &&
+	git fast-import <stream &&
+	git log -p --format="rev <%an, %ae>: %s" HEAD >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'author name and commit message' '
+	reinit_git &&
+	echo "<author@example.com, author@example.com@local>" >expect.author &&
+	cat >message <<-\EOF &&
+	A concise summary of the change
+
+	A detailed description of the change, why it is needed, what
+	was broken and why applying this is the best course of action.
+
+	* file.c
+	    Details pertaining to an individual file.
+	EOF
+	{
+		properties \
+			svn:author author@example.com \
+			svn:log "$(cat message)" &&
+		echo PROPS-END
+	} >props &&
+	{
+		echo "SVN-fs-dump-format-version: 3" &&
+		echo &&
+		echo "Revision-number: 1" &&
+		echo Prop-content-length: $(wc -c <props) &&
+		echo Content-length: $(wc -c <props) &&
+		echo &&
+		cat props
+	} >log.dump &&
+	test-svn-fe log.dump >stream &&
+	git fast-import <stream &&
+	git log -p --format="%B" HEAD >actual.log &&
+	git log --format="<%an, %ae>" >actual.author &&
+	test_cmp message actual.log &&
+	test_cmp expect.author actual.author
+'
+
+test_expect_success 'unsupported properties are ignored' '
+	reinit_git &&
+	echo author >expect &&
+	cat >extraprop.dump <<-\EOF &&
+	SVN-fs-dump-format-version: 3
+
+	Revision-number: 1
+	Prop-content-length: 56
+	Content-length: 56
+
+	K 8
+	nonsense
+	V 1
+	y
+	K 10
+	svn:author
+	V 6
+	author
+	PROPS-END
+	EOF
+	test-svn-fe extraprop.dump >stream &&
+	git fast-import <stream &&
+	git log -p --format=%an HEAD >actual &&
+	test_cmp expect actual
+'
+
+test_expect_failure 'timestamp and empty file' '
+	echo author@example.com >expect.author &&
+	echo 1999-01-01 >expect.date &&
+	echo file >expect.files &&
+	reinit_git &&
+	{
+		properties \
+			svn:author author@example.com \
+			svn:date "1999-01-01T00:01:002.000000Z" \
+			svn:log "add empty file" &&
+		echo PROPS-END
+	} >props &&
+	{
+		cat <<-EOF &&
+		SVN-fs-dump-format-version: 3
+
+		Revision-number: 1
+		EOF
+		echo Prop-content-length: $(wc -c <props) &&
+		echo Content-length: $(wc -c <props) &&
+		echo &&
+		cat props &&
+		cat <<-\EOF
+
+		Node-path: empty-file
+		Node-kind: file
+		Node-action: add
+		Content-length: 0
+
+		EOF
+	} >emptyfile.dump &&
+	test-svn-fe emptyfile.dump >stream &&
+	git fast-import <stream &&
+	git log --format=%an HEAD >actual.author &&
+	git log --date=short --format=%ad HEAD >actual.date &&
+	git ls-tree -r --name-only HEAD >actual.files &&
+	test_cmp expect.author actual.author &&
+	test_cmp expect.date actual.date &&
+	test_cmp expect.files actual.files &&
+	git checkout HEAD empty-file &&
+	test_cmp empty file
+'
+
+test_expect_success 'directory with files' '
+	reinit_git &&
+	printf "%s\n" directory/file1 directory/file2 >expect.files &&
+	echo hi >hi &&
+	echo hello >hello &&
+	{
+		properties \
+			svn:author author@example.com \
+			svn:date "1999-02-01T00:01:002.000000Z" \
+			svn:log "add directory with some files in it" &&
+		echo PROPS-END
+	} >props &&
+	{
+		cat <<-EOF &&
+		SVN-fs-dump-format-version: 3
+
+		Revision-number: 1
+		EOF
+		echo Prop-content-length: $(wc -c <props) &&
+		echo Content-length: $(wc -c <props) &&
+		echo &&
+		cat props &&
+		cat <<-\EOF &&
+
+		Node-path: directory
+		Node-kind: dir
+		Node-action: add
+		Prop-content-length: 10
+		Content-length: 10
+
+		PROPS-END
+
+		Node-path: directory/file1
+		Node-kind: file
+		Node-action: add
+		EOF
+		text_no_props hello &&
+		cat <<-\EOF &&
+		Node-path: directory/file2
+		Node-kind: file
+		Node-action: add
+		EOF
+		text_no_props hi
+	} >directory.dump &&
+	test-svn-fe directory.dump >stream &&
+	git fast-import <stream &&
+
+	git ls-tree -r --name-only HEAD >actual.files &&
+	git checkout HEAD directory &&
+	test_cmp expect.files actual.files &&
+	test_cmp hello directory/file1 &&
+	test_cmp hi directory/file2
+'
+
+test_expect_success 'deltas not supported' '
+	{
+		# (old) h + (inline) ello + (old) \n
+		printf "SVNQ%b%b%s" "Q\003\006\005\004" "\001Q\0204\001\002" "ello" |
+		q_to_nul
+	} >delta &&
+	{
+		properties \
+			svn:author author@example.com \
+			svn:date "1999-01-05T00:01:002.000000Z" \
+			svn:log "add greeting" &&
+		echo PROPS-END
+	} >props &&
+	{
+		properties \
+			svn:author author@example.com \
+			svn:date "1999-01-06T00:01:002.000000Z" \
+			svn:log "change it" &&
+		echo PROPS-END
+	} >props2 &&
+	{
+		echo SVN-fs-dump-format-version: 3 &&
+		echo &&
+		echo Revision-number: 1 &&
+		echo Prop-content-length: $(wc -c <props) &&
+		echo Content-length: $(wc -c <props) &&
+		echo &&
+		cat props &&
+		cat <<-\EOF &&
+
+		Node-path: hello
+		Node-kind: file
+		Node-action: add
+		Prop-content-length: 10
+		Text-content-length: 3
+		Content-length: 13
+
+		PROPS-END
+		hi
+
+		EOF
+		echo Revision-number: 2 &&
+		echo Prop-content-length: $(wc -c <props2) &&
+		echo Content-length: $(wc -c <props2) &&
+		echo &&
+		cat props2 &&
+		cat <<-\EOF &&
+
+		Node-path: hello
+		Node-kind: file
+		Node-action: change
+		Text-delta: true
+		Prop-content-length: 10
+		EOF
+		echo Text-content-length: $(wc -c <delta) &&
+		echo Content-length: $((10 + $(wc -c <delta))) &&
+		echo &&
+		echo PROPS-END &&
+		cat delta
+	} >delta.dump &&
+	test_must_fail test-svn-fe delta.dump
+'
+
+test_expect_success 'property deltas not supported' '
+	{
+		properties \
+			svn:author author@example.com \
+			svn:date "1999-03-06T00:01:002.000000Z" \
+			svn:log "make an executable, or chmod -x it" &&
+		echo PROPS-END
+	} >revprops &&
+	{
+		echo SVN-fs-dump-format-version: 3 &&
+		echo &&
+		echo Revision-number: 1 &&
+		echo Prop-content-length: $(wc -c <revprops) &&
+		echo Content-length: $(wc -c <revprops) &&
+		echo &&
+		cat revprops &&
+		echo &&
+		cat <<-\EOF &&
+		Node-path: script.sh
+		Node-kind: file
+		Node-action: add
+		Text-content-length: 0
+		Prop-content-length: 39
+		Content-length: 39
+
+		K 14
+		svn:executable
+		V 4
+		true
+		PROPS-END
+
+		EOF
+		echo Revision-number: 2 &&
+		echo Prop-content-length: $(wc -c <revprops) &&
+		echo Content-length: $(wc -c <revprops) &&
+		echo &&
+		cat revprops &&
+		echo &&
+		cat <<-\EOF
+		Node-path: script.sh
+		Node-kind: file
+		Node-action: change
+		Prop-delta: true
+		Prop-content-length: 30
+		Content-length: 30
+
+		D 14
+		svn:executable
+		PROPS-END
+		EOF
+	} >propdelta.dump &&
+	test_must_fail test-svn-fe propdelta.dump
+'
+
 test_expect_success 't9135/svn.dump' '
 	svnadmin create simple-svn &&
 	svnadmin load simple-svn <"$TEST_DIRECTORY/t9135/svn.dump" &&
diff --git a/vcs-svn/svndump.c b/vcs-svn/svndump.c
index fa580e6..6b64c1b 100644
--- a/vcs-svn/svndump.c
+++ b/vcs-svn/svndump.c
@@ -42,6 +42,7 @@ static char* log_copy(uint32_t length, char *log)
 static struct {
 	uint32_t action, propLength, textLength, srcRev, srcMode, mark, type;
 	uint32_t src[REPO_MAX_PATH_DEPTH], dst[REPO_MAX_PATH_DEPTH];
+	uint32_t text_delta, prop_delta;
 } node_ctx;
 
 static struct {
@@ -58,7 +59,9 @@ static struct {
 	uint32_t svn_log, svn_author, svn_date, svn_executable, svn_special, uuid,
 		revision_number, node_path, node_kind, node_action,
 		node_copyfrom_path, node_copyfrom_rev, text_content_length,
-		prop_content_length, content_length, svn_fs_dump_format_version;
+		prop_content_length, content_length, svn_fs_dump_format_version,
+		/* version 3 format */
+		text_delta, prop_delta;
 } keys;
 
 static void reset_node_ctx(char *fname)
@@ -72,6 +75,8 @@ static void reset_node_ctx(char *fname)
 	node_ctx.srcMode = 0;
 	pool_tok_seq(REPO_MAX_PATH_DEPTH, node_ctx.dst, "/", fname);
 	node_ctx.mark = 0;
+	node_ctx.text_delta = 0;
+	node_ctx.prop_delta = 0;
 }
 
 static void reset_rev_ctx(uint32_t revision)
@@ -107,6 +112,9 @@ static void init_keys(void)
 	keys.prop_content_length = pool_intern("Prop-content-length");
 	keys.content_length = pool_intern("Content-length");
 	keys.svn_fs_dump_format_version = pool_intern("SVN-fs-dump-format-version");
+	/* version 3 format (Subversion 1.1.0) */
+	keys.text_delta = pool_intern("Text-delta");
+	keys.prop_delta = pool_intern("Prop-delta");
 }
 
 static void read_props(void)
@@ -144,6 +152,9 @@ static void read_props(void)
 
 static void handle_node(void)
 {
+	if (node_ctx.text_delta || node_ctx.prop_delta)
+		die("text and property deltas not supported");
+
 	if (node_ctx.propLength != LENGTH_UNKNOWN && node_ctx.propLength)
 		read_props();
 
@@ -210,8 +221,8 @@ void svndump_read(const char *url)
 
 		if (key == keys.svn_fs_dump_format_version) {
 			dump_ctx.version = atoi(val);
-			if (dump_ctx.version > 2)
-				die("expected svn dump format version <= 2, found %d",
+			if (dump_ctx.version > 3)
+				die("expected svn dump format version <= 3, found %d",
 				    dump_ctx.version);
 		} else if (key == keys.uuid) {
 			dump_ctx.uuid = pool_intern(val);
@@ -255,6 +266,10 @@ void svndump_read(const char *url)
 			node_ctx.textLength = atoi(val);
 		} else if (key == keys.prop_content_length) {
 			node_ctx.propLength = atoi(val);
+		} else if (key == keys.text_delta) {
+			node_ctx.text_delta = !strcmp(val, "true");
+		} else if (key == keys.prop_delta) {
+			node_ctx.prop_delta = !strcmp(val, "true");
 		} else if (key == keys.content_length) {
 			len = atoi(val);
 			buffer_read_line();
-- 
1.7.2.3.551.g13682.dirty

  parent reply	other threads:[~2010-11-18  5:04 UTC|newest]

Thread overview: 40+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2010-11-18  5:00 [PATCH 0/2] svn-fe: recognize v3 dumps Jonathan Nieder
2010-11-18  5:02 ` [PATCH 1/2] vcs-svn: Error out for " Jonathan Nieder
2010-11-18  5:03 ` Jonathan Nieder [this message]
2010-11-20  0:45 ` [RFC/PATCH 0/15] svn-fe: support for property deltas (but not text deltas yet) Jonathan Nieder
2010-11-20  0:46   ` [PATCH 01/15] vcs-svn: Check for errors from open() Jonathan Nieder
2010-11-20  0:46   ` [PATCH 02/15] vcs-svn: Eliminate node_ctx.srcRev global Jonathan Nieder
2010-11-20  0:46   ` [PATCH 03/15] vcs-svn: Eliminate node_ctx.mark global Jonathan Nieder
2010-11-20  0:47   ` [PATCH 04/15] vcs-svn: Unclutter handle_node by introducing have_props var Jonathan Nieder
2010-11-20  0:48   ` [PATCH 05/15] vcs-svn: Use mark to indicate nodes with included text Jonathan Nieder
2010-11-20  0:49   ` [PATCH 06/15] vcs-svn: handle_node: Handle deletion case early Jonathan Nieder
2010-11-20  0:49   ` [PATCH 07/15] vcs-svn: Replace = Delete + Add Jonathan Nieder
2010-11-20  0:51   ` [PATCH 08/15] vcs-svn: Combine repo_replace and repo_modify functions Jonathan Nieder
2010-11-20  0:52   ` [PATCH 09/15] vcs-svn: Delay read of per-path properties Jonathan Nieder
2010-11-20  0:52   ` [PATCH 10/15] vcs-svn: Reject path nodes without Node-action Jonathan Nieder
2010-11-20 14:53     ` Jonathan Nieder
2010-11-20  0:53   ` [PATCH 11/15] vcs-svn: More dump format sanity checks Jonathan Nieder
2010-11-30 19:48     ` Jonathan Nieder
     [not found]       ` <20101205091605.GA4332@burratino>
2010-12-05  9:32         ` [PATCH 2/2] vcs-svn: fix intermittent repo_tree corruption Jonathan Nieder
2010-12-05  9:33       ` [PATCH jn/svn-fe-maint 0/2] " Jonathan Nieder
2010-12-05  9:35         ` [PATCH 1/2] treap: make treap_insert return inserted node Jonathan Nieder
2010-12-06 22:19     ` [PATCH jn/svn-fe] vcs-svn: Allow change nodes for root of tree (/) Jonathan Nieder
2010-12-06 23:12       ` Jonathan Nieder
2010-11-20  0:53   ` [PATCH 12/15] vcs-svn: Make source easier to read on small screens Jonathan Nieder
2010-11-20  0:54   ` [PATCH 13/15] vcs-svn: Split off function for handling of individual properties Jonathan Nieder
2010-11-20  0:54   ` [PATCH 14/15] vcs-svn: Sharpen parsing of property lines Jonathan Nieder
2010-11-20  0:57   ` [PATCH 15/15] vcs-svn: Implement Prop-delta handling Jonathan Nieder
2010-11-20 19:21   ` [WIP/PATCH 0/8] svn-fe: support for text deltas Jonathan Nieder
2010-11-20 19:22     ` [PATCH 1/8] svn-fe: Prepare for strbuf use Jonathan Nieder
2010-11-20 19:25     ` [PATCH 2/8] vcs-svn: Internal fast_export_save_blob helper Jonathan Nieder
2010-11-20 19:25     ` [PATCH 3/8] vcs-svn: Introduce repo_read_path to check the content at a path Jonathan Nieder
2011-03-06 12:29       ` Jonathan Nieder
2010-11-20 19:26     ` [PATCH 4/8] vcs-svn: Introduce fd_buffer routines Jonathan Nieder
2010-11-20 19:27     ` [PATCH 5/8] vcs-svn: Read delta preimage from file descriptor Jonathan Nieder
2010-11-20 19:28     ` [PATCH 6/8] vcs-svn: Let caller set up sliding window for delta preimage Jonathan Nieder
2010-11-20 19:31       ` Jonathan Nieder
2010-11-20 19:29     ` [PATCH 7/8] vcs-svn: Teach line_buffer about temporary files Jonathan Nieder
2010-11-20 19:29     ` [PATCH 8/8] vcs-svn: Implement text-delta handling Jonathan Nieder
2010-12-04 17:34       ` [PATCH 10/8] vcs-svn: Consume whole preimage when applying deltas Jonathan Nieder
2010-11-20 19:30     ` [PATCH 9/8] svn-fe: Test script for handling of dumps with --deltas Jonathan Nieder
2010-12-04 17:29       ` Jonathan Nieder

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20101118050351.GC14861@burratino \
    --to=jrnieder@gmail.com \
    --cc=artagnon@gmail.com \
    --cc=david.barr@cordelta.com \
    --cc=git@vger.kernel.org \
    --cc=srabbelier@gmail.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.