[PATCH v3 00/10] vcs-svn,svn-fe add a couple of options

[PATCH v3 00/10] vcs-svn,svn-fe add a couple of options

[Dmitry Ivankov]

This is a next iteration of [1].

Most Noticeable changes since [1] are:
1) squash [2/11] and [4/11] into [3/10] "use parse-options"
2) replace [11/11] with a lesser patch that just adds api for
writing notes in vcs-svn/ but doesn't yet uses it. So that the
series is more stable, while exact notes usage can be adjusted
and played with a bit more separately.
3) applied most of suggestions from [1] with minor edits

The patch base is svn-fe branch at git://repo.or.cz/git/jrn.git

[1] http://thread.gmane.org/gmane.comp.version-control.git/177025

Dmitry Ivankov (10):
  svn-fe: add man target to Makefile
  svn-fe: add EXTLIBS needed for parse-options
  svn-fe,test-svn-fe: use parse-options
  vcs-svn: make svndump_init parameters a struct
  vcs-svn: move commit parameters logic to svndump.c
  vcs-svn,svn-fe: allow to specify dump destination ref
  vcs-svn,svn-fe: convert REPORT_FILENO to an option
  vcs-svn,svn-fe: allow to disable 'progress' lines
  vcs-svn,svn-fe: add --incremental option
  vcs-svn: add fast_export_note to create notes

 contrib/svn-fe/Makefile   |   18 ++++----
 contrib/svn-fe/svn-fe.c   |   45 ++++++++++++++++--
 contrib/svn-fe/svn-fe.txt |   38 +++++++++++++--
 t/t9010-svn-fe.sh         |  118 ++++++++++++++++++++++++++++++++++++++-------
 test-svn-fe.c             |   57 ++++++++++++++++------
 vcs-svn/fast_export.c     |   60 ++++++++++-------------
 vcs-svn/fast_export.h     |   10 ++--
 vcs-svn/svndump.c         |   79 ++++++++++++++++++++++++------
 vcs-svn/svndump.h         |   14 +++++-
 9 files changed, 332 insertions(+), 107 deletions(-)

-- 
1.7.3.4

[PATCH v3 01/10] svn-fe: add man target to Makefile

[Dmitry Ivankov]

There already is a svn-fe.1 target. But 'man' being a standard
target is easier to discover or type. It can also be reused if
more manpages arise here.

Signed-off-by: Dmitry Ivankov <divanorama@gmail.com>
---
 contrib/svn-fe/Makefile |    4 +++-
 1 files changed, 3 insertions(+), 1 deletions(-)

diff --git a/contrib/svn-fe/Makefile b/contrib/svn-fe/Makefile
index 360d8da..bc03a3e 100644
--- a/contrib/svn-fe/Makefile
+++ b/contrib/svn-fe/Makefile
@@ -33,6 +33,8 @@ ifndef V
 endif
 endif
 
+man: svn-fe.1
+
 svn-fe$X: svn-fe.o $(VCSSVN_LIB) $(GIT_LIB)
 	$(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $@ svn-fe.o \
 		$(ALL_LDFLAGS) $(LIBS)
@@ -60,4 +62,4 @@ svn-fe.1: svn-fe.txt
 clean:
 	$(RM) svn-fe$X svn-fe.o svn-fe.html svn-fe.xml svn-fe.1
 
-.PHONY: all clean FORCE
+.PHONY: all man clean FORCE
-- 
1.7.3.4

[PATCH v3 02/10] svn-fe: add EXTLIBS needed for parse-options

[Dmitry Ivankov]

Currently parse-options.o pull quite a big bunch of dependencies
that are neither pulled in by svn-fe Makefile nor included in libgit.a.

Use a temporary hack: put hardcoded EXTLIBS, this may not work in all
setups because /Makefile logic is not repeated. The list of extlibs
is likely to be exhaustive, but one may need to adjust it.

-lcrypto is needed for SHA-1 routines unless NO_OPENSSL or BLK_SHA1
is set, -lpcre is for grep if USE_LIBPCRE is set, and -lz is needed
throughout.

In the future, none of these should be needed, after a little
rearranging to ensure that parse-options.o has no references to
translation units that need to access the object db.

Signed-off-by: Dmitry Ivankov <divanorama@gmail.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 contrib/svn-fe/Makefile |   12 +++++-------
 1 files changed, 5 insertions(+), 7 deletions(-)

diff --git a/contrib/svn-fe/Makefile b/contrib/svn-fe/Makefile
index bc03a3e..8b12df1 100644
--- a/contrib/svn-fe/Makefile
+++ b/contrib/svn-fe/Makefile
@@ -8,11 +8,12 @@ CFLAGS = -g -O2 -Wall
 LDFLAGS =
 ALL_CFLAGS = $(CFLAGS)
 ALL_LDFLAGS = $(LDFLAGS)
-EXTLIBS =
+EXTLIBS = -lssl -lcrypto -lpcre -lz -lpthread
 
 GIT_LIB = ../../libgit.a
 VCSSVN_LIB = ../../vcs-svn/lib.a
-LIBS = $(VCSSVN_LIB) $(GIT_LIB) $(EXTLIBS)
+XDIFF_LIB = ../../xdiff/lib.a
+LIBS = $(VCSSVN_LIB) $(GIT_LIB) $(XDIFF_LIB) $(EXTLIBS)
 
 QUIET_SUBDIR0 = +$(MAKE) -C # space to separate -C and subdir
 QUIET_SUBDIR1 =
@@ -53,11 +54,8 @@ svn-fe.1: svn-fe.txt
 		../contrib/svn-fe/$@
 	$(MV) ../../Documentation/svn-fe.1 .
 
-../../vcs-svn/lib.a: FORCE
-	$(QUIET_SUBDIR0)../.. $(QUIET_SUBDIR1) vcs-svn/lib.a
-
-../../libgit.a: FORCE
-	$(QUIET_SUBDIR0)../.. $(QUIET_SUBDIR1) libgit.a
+$(VCSSVN_LIB) $(GIT_LIB) $(XDIFF_LIB): ../../%.a: FORCE
+	$(QUIET_SUBDIR0)../.. $(QUIET_SUBDIR1) $*.a
 
 clean:
 	$(RM) svn-fe$X svn-fe.o svn-fe.html svn-fe.xml svn-fe.1
-- 
1.7.3.4

[PATCH v3 03/10] svn-fe,test-svn-fe: use parse-options

[Dmitry Ivankov]

There was custom options parsing. As more options arise it will
be easier to add and document new options with parse-options api.

Use parse-options api in svn-fe and test-svn-fe. This implies adding
help messages. And by the way clarify the "url" parameter meaning,
renaming it to git-svn-id-url and updating svn-fe.txt. Also allow a
--git-svn-id-url=url way of specifying it.

$ svn-fe --git-svn-id-url=url
does the same thing as
$ svn-fe url
i.e., url is used to generate git-svn-id: lines, if url is set.

Signed-off-by: Dmitry Ivankov <divanorama@gmail.com>
Reviewed-by: Jonathan Nieder <jrnieder@gmail.com>
---
 contrib/svn-fe/Makefile   |    2 +-
 contrib/svn-fe/svn-fe.c   |   32 +++++++++++++++++++++++++++++---
 contrib/svn-fe/svn-fe.txt |   17 +++++++++++++----
 test-svn-fe.c             |   43 +++++++++++++++++++++++++++++--------------
 4 files changed, 72 insertions(+), 22 deletions(-)

diff --git a/contrib/svn-fe/Makefile b/contrib/svn-fe/Makefile
index 8b12df1..15ba24d 100644
--- a/contrib/svn-fe/Makefile
+++ b/contrib/svn-fe/Makefile
@@ -41,7 +41,7 @@ svn-fe$X: svn-fe.o $(VCSSVN_LIB) $(GIT_LIB)
 		$(ALL_LDFLAGS) $(LIBS)
 
 svn-fe.o: svn-fe.c ../../vcs-svn/svndump.h
-	$(QUIET_CC)$(CC) -I../../vcs-svn -o $*.o -c $(ALL_CFLAGS) $<
+	$(QUIET_CC)$(CC) -I../../vcs-svn -I../.. -o $*.o -c $(ALL_CFLAGS) $<
 
 svn-fe.html: svn-fe.txt
 	$(QUIET_SUBDIR0)../../Documentation $(QUIET_SUBDIR1) \
diff --git a/contrib/svn-fe/svn-fe.c b/contrib/svn-fe/svn-fe.c
index 35db24f..a95e72f 100644
--- a/contrib/svn-fe/svn-fe.c
+++ b/contrib/svn-fe/svn-fe.c
@@ -3,14 +3,40 @@
  * You may freely use, modify, distribute, and relicense it.
  */
 
-#include <stdlib.h>
+#include "git-compat-util.h"
+#include "parse-options.h"
 #include "svndump.h"
 
-int main(int argc, char **argv)
+static const char * const svn_fe_usage[] = {
+	"svn-fe [options] [git-svn-id-url] < dump | fast-import-backend",
+	NULL
+};
+
+static const char *url;
+
+static struct option svn_fe_options[] = {
+	OPT_STRING(0, "git-svn-id-url", &url, "url",
+		"add git-svn-id line to log messages, imitating git-svn"),
+	OPT_END()
+};
+
+int main(int argc, const char **argv)
 {
+	argc = parse_options(argc, argv, NULL, svn_fe_options,
+						svn_fe_usage, 0);
+	if (argc > 1)
+		usage_with_options(svn_fe_usage, svn_fe_options);
+
+	if (argc == 1) {
+		if (url)
+			usage_msg_opt("git-svn-id-url is set twice: as a "
+					"--parameter and as a [parameter]",
+					svn_fe_usage, svn_fe_options);
+		url = argv[0];
+	}
 	if (svndump_init(NULL))
 		return 1;
-	svndump_read((argc > 1) ? argv[1] : NULL);
+	svndump_read(url);
 	svndump_deinit();
 	svndump_reset();
 	return 0;
diff --git a/contrib/svn-fe/svn-fe.txt b/contrib/svn-fe/svn-fe.txt
index 2dd27ce..8c6d347 100644
--- a/contrib/svn-fe/svn-fe.txt
+++ b/contrib/svn-fe/svn-fe.txt
@@ -10,7 +10,7 @@ SYNOPSIS
 [verse]
 mkfifo backchannel &&
 svnadmin dump --deltas REPO |
-	svn-fe [url] 3<backchannel |
+	svn-fe [options] [git-svn-id-url] 3<backchannel |
 	git fast-import --cat-blob-fd=3 3>backchannel
 
 DESCRIPTION
@@ -25,6 +25,14 @@ command.
 Note: this tool is very young.  The details of its commandline
 interface may change in backward incompatible ways.
 
+OPTIONS
+-------
+
+--git-svn-id-url=<url>::
+	Url to be used in git-svn-id: lines in git-svn
+	metadata lines format. See NOTES for more detailed
+	description.
+
 INPUT FORMAT
 ------------
 Subversion's repository dump format is documented in full in
@@ -50,9 +58,10 @@ user <user@UUID>
 as committer, where 'user' is the value of the `svn:author` property
 and 'UUID' the repository's identifier.
 
-To support incremental imports, 'svn-fe' puts a `git-svn-id` line at
-the end of each commit log message if passed an url on the command
-line.  This line has the form `git-svn-id: URL@REVNO UUID`.
+'svn-fe' can be used in preparing a repository for 'git-svn' as follows.
+If `git-svn-id-url` is specified, 'svn-fe' will put `git-svn-id` line at
+the end of each commit log message.
+This line has the form `git-svn-id: URL@REVNO UUID`.
 
 The resulting repository will generally require further processing
 to put each project in its own repository and to separate the history
diff --git a/test-svn-fe.c b/test-svn-fe.c
index 332a5f7..c10d3ca 100644
--- a/test-svn-fe.c
+++ b/test-svn-fe.c
@@ -3,28 +3,39 @@
  */
 
 #include "git-compat-util.h"
+#include "parse-options.h"
 #include "vcs-svn/svndump.h"
 #include "vcs-svn/svndiff.h"
 #include "vcs-svn/sliding_window.h"
 #include "vcs-svn/line_buffer.h"
 
-static const char test_svnfe_usage[] =
-	"test-svn-fe (<dumpfile> | [-d] <preimage> <delta> <len>)";
+static const char * const test_svnfe_usage[] = {
+	"test-svn-fe <dumpfile>",
+	"test-svn-fe -d <preimage> <delta> <len>",
+	NULL
+};
 
-static int apply_delta(int argc, char *argv[])
+static int delta_test;
+
+static struct option test_svnfe_options[] = {
+	OPT_SET_INT('d', "apply-delta", &delta_test, "test apply_delta", 1),
+	OPT_END()
+};
+
+static int apply_delta(int argc, const char *argv[])
 {
 	struct line_buffer preimage = LINE_BUFFER_INIT;
 	struct line_buffer delta = LINE_BUFFER_INIT;
 	struct sliding_view preimage_view = SLIDING_VIEW_INIT(&preimage, -1);
 
-	if (argc != 5)
-		usage(test_svnfe_usage);
+	if (argc != 3)
+		usage_with_options(test_svnfe_usage, test_svnfe_options);
 
-	if (buffer_init(&preimage, argv[2]))
+	if (buffer_init(&preimage, argv[0]))
 		die_errno("cannot open preimage");
-	if (buffer_init(&delta, argv[3]))
+	if (buffer_init(&delta, argv[1]))
 		die_errno("cannot open delta");
-	if (svndiff0_apply(&delta, (off_t) strtoull(argv[4], NULL, 0),
+	if (svndiff0_apply(&delta, (off_t) strtoull(argv[2], NULL, 0),
 					&preimage_view, stdout))
 		return 1;
 	if (buffer_deinit(&preimage))
@@ -37,10 +48,16 @@ static int apply_delta(int argc, char *argv[])
 	return 0;
 }
 
-int main(int argc, char *argv[])
+int main(int argc, const char *argv[])
 {
-	if (argc == 2) {
-		if (svndump_init(argv[1]))
+	argc = parse_options(argc, argv, NULL, test_svnfe_options,
+						test_svnfe_usage, 0);
+
+	if (delta_test)
+		return apply_delta(argc, argv);
+
+	if (argc == 1) {
+		if (svndump_init(argv[0]))
 			return 1;
 		svndump_read(NULL);
 		svndump_deinit();
@@ -48,7 +65,5 @@ int main(int argc, char *argv[])
 		return 0;
 	}
 
-	if (argc >= 2 && !strcmp(argv[1], "-d"))
-		return apply_delta(argc, argv);
-	usage(test_svnfe_usage);
+	usage_with_options(test_svnfe_usage, test_svnfe_options);
 }
-- 
1.7.3.4

[PATCH v3 04/10] vcs-svn: make svndump_init parameters a struct

[Dmitry Ivankov]

svndump_init takes a dumpfile parameter and svndump_read takes url
parameter. Internally url is stored in dump_ctx that is in fact
reset in svndump_init, but then is reset once more in svndump_read.
It'd be better to make url a svndump_init parameter and avoid this
double reset.

A bunch of new svndump parameters are going to be introduced. So wrap
them all to a svndump_options struct to make adding new ones smooth
and easy.

The usage changes like following.
Before:
	if (svndump_init(dumpfile))
		die("svndump_init failed");
	svndump_read(url);
After:
	struct svndump_args opts;
	memset(&opts, 0, sizeof(opts));
	opts.url = url;
	opts.filename = dumpfile;
	if (svndump_init(&opts))
		die("svndump_init failed");
	svndump_read();

Signed-off-by: Dmitry Ivankov <divanorama@gmail.com>
---
 contrib/svn-fe/svn-fe.c |   12 ++++++------
 test-svn-fe.c           |    7 +++++--
 vcs-svn/svndump.c       |   12 ++++++------
 vcs-svn/svndump.h       |   11 +++++++++--
 4 files changed, 26 insertions(+), 16 deletions(-)

diff --git a/contrib/svn-fe/svn-fe.c b/contrib/svn-fe/svn-fe.c
index a95e72f..0165c3f 100644
--- a/contrib/svn-fe/svn-fe.c
+++ b/contrib/svn-fe/svn-fe.c
@@ -12,10 +12,10 @@ static const char * const svn_fe_usage[] = {
 	NULL
 };
 
-static const char *url;
+static struct svndump_options options;
 
 static struct option svn_fe_options[] = {
-	OPT_STRING(0, "git-svn-id-url", &url, "url",
+	OPT_STRING(0, "git-svn-id-url", &options.git_svn_url, "url",
 		"add git-svn-id line to log messages, imitating git-svn"),
 	OPT_END()
 };
@@ -28,15 +28,15 @@ int main(int argc, const char **argv)
 		usage_with_options(svn_fe_usage, svn_fe_options);
 
 	if (argc == 1) {
-		if (url)
+		if (options.git_svn_url)
 			usage_msg_opt("git-svn-id-url is set twice: as a "
 					"--parameter and as a [parameter]",
 					svn_fe_usage, svn_fe_options);
-		url = argv[0];
+		options.git_svn_url = argv[0];
 	}
-	if (svndump_init(NULL))
+	if (svndump_init(&options))
 		return 1;
-	svndump_read(url);
+	svndump_read();
 	svndump_deinit();
 	svndump_reset();
 	return 0;
diff --git a/test-svn-fe.c b/test-svn-fe.c
index c10d3ca..0dd0657 100644
--- a/test-svn-fe.c
+++ b/test-svn-fe.c
@@ -15,6 +15,8 @@ static const char * const test_svnfe_usage[] = {
 	NULL
 };
 
+static struct svndump_options options;
+
 static int delta_test;
 
 static struct option test_svnfe_options[] = {
@@ -57,9 +59,10 @@ int main(int argc, const char *argv[])
 		return apply_delta(argc, argv);
 
 	if (argc == 1) {
-		if (svndump_init(argv[0]))
+		options.dumpfile = argv[0];
+		if (svndump_init(&options))
 			return 1;
-		svndump_read(NULL);
+		svndump_read();
 		svndump_deinit();
 		svndump_reset();
 		return 0;
diff --git a/vcs-svn/svndump.c b/vcs-svn/svndump.c
index b1f4161..5cdf6b8 100644
--- a/vcs-svn/svndump.c
+++ b/vcs-svn/svndump.c
@@ -12,6 +12,7 @@
 #include "fast_export.h"
 #include "line_buffer.h"
 #include "strbuf.h"
+#include "svndump.h"
 
 /*
  * Compare start of string to literal of equal length;
@@ -313,14 +314,13 @@ static void end_revision(void)
 		fast_export_end_commit(rev_ctx.revision);
 }
 
-void svndump_read(const char *url)
+void svndump_read(void)
 {
 	char *val;
 	char *t;
 	uint32_t active_ctx = DUMP_CTX;
 	uint32_t len;
 
-	reset_dump_ctx(url);
 	while ((t = buffer_read_line(&input))) {
 		val = strchr(t, ':');
 		if (!val)
@@ -455,10 +455,10 @@ void svndump_read(const char *url)
 		end_revision();
 }
 
-int svndump_init(const char *filename)
+int svndump_init(const struct svndump_options *o)
 {
-	if (buffer_init(&input, filename))
-		return error("cannot open %s: %s", filename, strerror(errno));
+	if (buffer_init(&input, o->dumpfile))
+		return error("cannot open %s: %s", o->dumpfile, strerror(errno));
 	fast_export_init(REPORT_FILENO);
 	strbuf_init(&dump_ctx.uuid, 4096);
 	strbuf_init(&dump_ctx.url, 4096);
@@ -466,7 +466,7 @@ int svndump_init(const char *filename)
 	strbuf_init(&rev_ctx.author, 4096);
 	strbuf_init(&node_ctx.src, 4096);
 	strbuf_init(&node_ctx.dst, 4096);
-	reset_dump_ctx(NULL);
+	reset_dump_ctx(o->git_svn_url);
 	reset_rev_ctx(0);
 	reset_node_ctx(NULL);
 	return 0;
diff --git a/vcs-svn/svndump.h b/vcs-svn/svndump.h
index df9ceb0..db39dfe 100644
--- a/vcs-svn/svndump.h
+++ b/vcs-svn/svndump.h
@@ -1,8 +1,15 @@
 #ifndef SVNDUMP_H_
 #define SVNDUMP_H_
 
-int svndump_init(const char *filename);
-void svndump_read(const char *url);
+struct svndump_options {
+	/*
+	 * dumpfile is opened in svndump_init and is read in svndump_read.
+	 */
+	const char *dumpfile, *git_svn_url;
+};
+
+int svndump_init(const struct svndump_options *o);
+void svndump_read(void);
 void svndump_deinit(void);
 void svndump_reset(void);
 
-- 
1.7.3.4

[PATCH v3 05/10] vcs-svn: move commit parameters logic to svndump.c

[Dmitry Ivankov]

fast_export.c had logic to set up commit ref, author name, email,
parent commit, import mark and git-svn-id: line based on both it's
own state (current import batch history) and the arguments passed.

Do separate the layers: make fast_export focus on producing the
fast-import stream, applying the deltas but not on svn-fe specific
logic. svndump now is responsible for choosing commit parents, marks,
ref name. Making it possible to generate incremental streams, produce
stream for several branches at a time, customize progress lines generation
and adding new logic becomes easier.

fast_export API changes:
- make fast_export_begin_commit to be more intuitive by using a set of
  parameters closer to what gets written to fast-import.

- rename fast_export_end_commit to fast_export_progress as it does only
  a progress line generation. fast_export_end_commit can be reintroduced
  once the need will arise.

- git-svn-id line is now a caller concern. If it is needed, it should be
  simply appended to the log message.

- author_name and author_email are now generated by the caller.

- ref name to be updated with the commit is now a parameter rather than
  a fixed "refs/heads/master".

The caller now may have to setup temporary buffers for author identity,
ref names, etc. This is a small additional per-commit cost to arrange
and/or copy them. Though, it's only per-commit rather that per-path
and might be worth the gain in readablity.

Signed-off-by: Dmitry Ivankov <divanorama@gmail.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 vcs-svn/fast_export.c |   47 +++++++++++++----------------------------------
 vcs-svn/fast_export.h |    8 ++++----
 vcs-svn/svndump.c     |   44 +++++++++++++++++++++++++++++++++++++++-----
 3 files changed, 56 insertions(+), 43 deletions(-)

diff --git a/vcs-svn/fast_export.c b/vcs-svn/fast_export.c
index 19d7c34..3dfccd2 100644
--- a/vcs-svn/fast_export.c
+++ b/vcs-svn/fast_export.c
@@ -13,9 +13,6 @@
 #include "sliding_window.h"
 #include "line_buffer.h"
 
-#define MAX_GITSVN_LINE_LEN 4096
-
-static uint32_t first_commit_done;
 static struct line_buffer postimage = LINE_BUFFER_INIT;
 static struct line_buffer report_buffer = LINE_BUFFER_INIT;
 
@@ -31,7 +28,6 @@ static int init_postimage(void)
 
 void fast_export_init(int fd)
 {
-	first_commit_done = 0;
 	if (buffer_fdinit(&report_buffer, fd))
 		die_errno("cannot read from file descriptor %d", fd);
 }
@@ -73,40 +69,23 @@ void fast_export_modify(const char *path, uint32_t mode, const char *dataref)
 	putchar('\n');
 }
 
-static char gitsvnline[MAX_GITSVN_LINE_LEN];
-void fast_export_begin_commit(uint32_t revision, const char *author,
-			const struct strbuf *log,
-			const char *uuid, const char *url,
-			unsigned long timestamp)
+void fast_export_begin_commit(const char *ref, uint32_t mark, const char *from,
+			const char *author_name, const char *author_email,
+			const struct strbuf *log, unsigned long timestamp)
 {
-	static const struct strbuf empty = STRBUF_INIT;
-	if (!log)
-		log = &empty;
-	if (*uuid && *url) {
-		snprintf(gitsvnline, MAX_GITSVN_LINE_LEN,
-				"\n\ngit-svn-id: %s@%"PRIu32" %s\n",
-				 url, revision, uuid);
-	} else {
-		*gitsvnline = '\0';
-	}
-	printf("commit refs/heads/master\n");
-	printf("mark :%"PRIu32"\n", revision);
-	printf("committer %s <%s@%s> %ld +0000\n",
-		   *author ? author : "nobody",
-		   *author ? author : "nobody",
-		   *uuid ? uuid : "local", timestamp);
-	printf("data %"PRIuMAX"\n",
-		(uintmax_t) (log->len + strlen(gitsvnline)));
+	printf("commit %s\n", ref);
+	if (mark)
+		printf("mark :%"PRIu32"\n", mark);
+	printf("committer %s <%s> %ld +0000\n",
+		author_name, author_email, timestamp);
+	printf("data %"PRIuMAX"\n", (uintmax_t) log->len);
 	fwrite(log->buf, log->len, 1, stdout);
-	printf("%s\n", gitsvnline);
-	if (!first_commit_done) {
-		if (revision > 1)
-			printf("from :%"PRIu32"\n", revision - 1);
-		first_commit_done = 1;
-	}
+	putchar('\n');
+	if (from && *from)
+		printf("from %s\n", from);
 }
 
-void fast_export_end_commit(uint32_t revision)
+void fast_export_progress(uint32_t revision)
 {
 	printf("progress Imported commit %"PRIu32".\n\n", revision);
 }
diff --git a/vcs-svn/fast_export.h b/vcs-svn/fast_export.h
index 43d05b6..bf58880 100644
--- a/vcs-svn/fast_export.h
+++ b/vcs-svn/fast_export.h
@@ -10,10 +10,10 @@ void fast_export_reset(void);
 
 void fast_export_delete(const char *path);
 void fast_export_modify(const char *path, uint32_t mode, const char *dataref);
-void fast_export_begin_commit(uint32_t revision, const char *author,
-			const struct strbuf *log, const char *uuid,
-			const char *url, unsigned long timestamp);
-void fast_export_end_commit(uint32_t revision);
+void fast_export_begin_commit(const char *ref, uint32_t mark, const char *from,
+			const char *author_name, const char *author_email,
+			const struct strbuf *log, unsigned long timestamp);
+void fast_export_progress(uint32_t revision);
 void fast_export_data(uint32_t mode, uint32_t len, struct line_buffer *input);
 void fast_export_blob_delta(uint32_t mode,
 			uint32_t old_mode, const char *old_data,
diff --git a/vcs-svn/svndump.c b/vcs-svn/svndump.c
index 5cdf6b8..28d84c9 100644
--- a/vcs-svn/svndump.c
+++ b/vcs-svn/svndump.c
@@ -37,6 +37,8 @@
 #define LENGTH_UNKNOWN (~0)
 #define DATE_RFC2822_LEN 31
 
+#define MAX_GITSVN_LINE_LEN 4096
+
 static struct line_buffer input = LINE_BUFFER_INIT;
 
 static struct {
@@ -54,6 +56,7 @@ static struct {
 static struct {
 	uint32_t version;
 	struct strbuf uuid, url;
+	int first_commit_done;
 } dump_ctx;
 
 static void reset_node_ctx(char *fname)
@@ -86,6 +89,7 @@ static void reset_dump_ctx(const char *url)
 		strbuf_addstr(&dump_ctx.url, url);
 	dump_ctx.version = 1;
 	strbuf_reset(&dump_ctx.uuid);
+	dump_ctx.first_commit_done = 0;
 }
 
 static void handle_property(const struct strbuf *key_buf,
@@ -299,19 +303,49 @@ static void handle_node(void)
 				node_ctx.textLength, &input);
 }
 
+static void add_metadata_trailer(struct strbuf *buf)
+{
+	if (*dump_ctx.uuid.buf && *dump_ctx.url.buf)
+		strbuf_addf(buf, "\n\ngit-svn-id: %s@%"PRIu32" %s\n",
+			 dump_ctx.url.buf, rev_ctx.revision, dump_ctx.uuid.buf);
+}
+
 static void begin_revision(void)
 {
+	static struct strbuf email;
+	const char *author;
+	uint32_t prev;
+	char buf[32];
+
 	if (!rev_ctx.revision)	/* revision 0 gets no git commit. */
 		return;
-	fast_export_begin_commit(rev_ctx.revision, rev_ctx.author.buf,
-		&rev_ctx.log, dump_ctx.uuid.buf, dump_ctx.url.buf,
-		rev_ctx.timestamp);
+	prev = dump_ctx.first_commit_done ? rev_ctx.revision - 1 : 0;
+	if (prev)
+		snprintf(buf, 32, ":%"PRIu32, prev);
+	else
+		*buf = 0;
+	author = *rev_ctx.author.buf ? rev_ctx.author.buf : "nobody";
+
+	strbuf_reset(&email);
+	strbuf_addstr(&email, author);
+	strbuf_addch(&email, '@');
+	if (*dump_ctx.uuid.buf)
+		strbuf_addstr(&email, dump_ctx.uuid.buf);
+	else
+		strbuf_addstr(&email, "local");
+
+	add_metadata_trailer(&rev_ctx.log);
+
+	fast_export_begin_commit("refs/heads/master", rev_ctx.revision, buf,
+		author, email.buf, &rev_ctx.log, rev_ctx.timestamp);
 }
 
 static void end_revision(void)
 {
-	if (rev_ctx.revision)
-		fast_export_end_commit(rev_ctx.revision);
+	if (rev_ctx.revision) {
+		fast_export_progress(rev_ctx.revision);
+		dump_ctx.first_commit_done = 1;
+	}
 }
 
 void svndump_read(void)
-- 
1.7.3.4

[PATCH v3 06/10] vcs-svn,svn-fe: allow to specify dump destination ref

[Dmitry Ivankov]

svn-fe produces fast-import stream for a fixed refs/heads/master ref.
It is usually desired to write to a different ref. In a remote helper
it would be a ref in private namespace. If svn-fe is used by someone
directly it'll be more safe to remind where the commits can go. And
in both cases it may be needed to import from two repos and hence to
different refs.

Add a destination ref parameter to vcs-svn/, a corresponding parameter
to svn-fe and a simple test for it.

$ svn-fe --ref=refs/heads/master ...
is an explicit way to stay with the default destination.

Signed-off-by: Dmitry Ivankov <divanorama@gmail.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 contrib/svn-fe/svn-fe.c   |    2 ++
 contrib/svn-fe/svn-fe.txt |    4 ++++
 t/t9010-svn-fe.sh         |   42 ++++++++++++++++++++++++++----------------
 test-svn-fe.c             |    4 +++-
 vcs-svn/svndump.c         |   16 +++++++++++-----
 vcs-svn/svndump.h         |    1 +
 6 files changed, 47 insertions(+), 22 deletions(-)

diff --git a/contrib/svn-fe/svn-fe.c b/contrib/svn-fe/svn-fe.c
index 0165c3f..9dd8336 100644
--- a/contrib/svn-fe/svn-fe.c
+++ b/contrib/svn-fe/svn-fe.c
@@ -17,6 +17,8 @@ static struct svndump_options options;
 static struct option svn_fe_options[] = {
 	OPT_STRING(0, "git-svn-id-url", &options.git_svn_url, "url",
 		"add git-svn-id line to log messages, imitating git-svn"),
+	OPT_STRING(0, "ref", &options.ref, "refname",
+		"write to <refname> instead of refs/heads/master"),
 	OPT_END()
 };
 
diff --git a/contrib/svn-fe/svn-fe.txt b/contrib/svn-fe/svn-fe.txt
index 8c6d347..0d19475 100644
--- a/contrib/svn-fe/svn-fe.txt
+++ b/contrib/svn-fe/svn-fe.txt
@@ -33,6 +33,10 @@ OPTIONS
 	metadata lines format. See NOTES for more detailed
 	description.
 
+--ref=<refname>::
+	Ref to be written by the generated stream.
+	Default is refs/heads/master.
+
 INPUT FORMAT
 ------------
 Subversion's repository dump format is documented in full in
diff --git a/t/t9010-svn-fe.sh b/t/t9010-svn-fe.sh
index b7eed24..b45527e 100755
--- a/t/t9010-svn-fe.sh
+++ b/t/t9010-svn-fe.sh
@@ -20,9 +20,10 @@ try_dump () {
 	input=$1 &&
 	maybe_fail_svnfe=${2:+test_$2} &&
 	maybe_fail_fi=${3:+test_$3} &&
+	args=${4:-} &&
 
 	{
-		$maybe_fail_svnfe test-svn-fe "$input" >stream 3<backflow &
+		$maybe_fail_svnfe test-svn-fe $args "$input" >stream 3<backflow &
 	} &&
 	$maybe_fail_fi git fast-import --cat-blob-fd=3 <stream 3>backflow &&
 	wait $!
@@ -54,6 +55,22 @@ text_no_props () {
 
 >empty
 
+cat >emptyprop.dump <<\EOF
+SVN-fs-dump-format-version: 3
+
+Revision-number: 1
+Prop-content-length: 10
+Content-length: 10
+
+PROPS-END
+
+Revision-number: 2
+Prop-content-length: 10
+Content-length: 10
+
+PROPS-END
+EOF
+
 test_expect_success 'setup: have pipes?' '
 	rm -f frob &&
 	if mkfifo frob
@@ -97,26 +114,19 @@ test_expect_failure PIPE 'empty revision' '
 test_expect_success PIPE 'empty properties' '
 	reinit_git &&
 	printf "rev <nobody, nobody@local>: %s\n" "" "" >expect &&
-	cat >emptyprop.dump <<-\EOF &&
-	SVN-fs-dump-format-version: 3
-
-	Revision-number: 1
-	Prop-content-length: 10
-	Content-length: 10
-
-	PROPS-END
-
-	Revision-number: 2
-	Prop-content-length: 10
-	Content-length: 10
-
-	PROPS-END
-	EOF
 	try_dump emptyprop.dump &&
 	git log -p --format="rev <%an, %ae>: %s" HEAD >actual &&
 	test_cmp expect actual
 '
 
+test_expect_success PIPE 'import to notmaster ref' '
+	reinit_git &&
+	try_dump emptyprop.dump "" "" "--ref=refs/heads/notmaster" &&
+
+	git rev-parse --verify notmaster &&
+	test_must_fail git rev-parse --verify master
+'
+
 test_expect_success PIPE 'author name and commit message' '
 	reinit_git &&
 	echo "<author@example.com, author@example.com@local>" >expect.author &&
diff --git a/test-svn-fe.c b/test-svn-fe.c
index 0dd0657..fddd3e8 100644
--- a/test-svn-fe.c
+++ b/test-svn-fe.c
@@ -10,7 +10,7 @@
 #include "vcs-svn/line_buffer.h"
 
 static const char * const test_svnfe_usage[] = {
-	"test-svn-fe <dumpfile>",
+	"test-svn-fe [options] <dumpfile>",
 	"test-svn-fe -d <preimage> <delta> <len>",
 	NULL
 };
@@ -21,6 +21,8 @@ static int delta_test;
 
 static struct option test_svnfe_options[] = {
 	OPT_SET_INT('d', "apply-delta", &delta_test, "test apply_delta", 1),
+	OPT_STRING(0, "ref", &options.ref, "refname",
+		"write to <refname> instead of refs/heads/master"),
 	OPT_END()
 };
 
diff --git a/vcs-svn/svndump.c b/vcs-svn/svndump.c
index 28d84c9..3a64708 100644
--- a/vcs-svn/svndump.c
+++ b/vcs-svn/svndump.c
@@ -57,6 +57,7 @@ static struct {
 	uint32_t version;
 	struct strbuf uuid, url;
 	int first_commit_done;
+	struct strbuf ref_name;
 } dump_ctx;
 
 static void reset_node_ctx(char *fname)
@@ -82,7 +83,7 @@ static void reset_rev_ctx(uint32_t revision)
 	strbuf_reset(&rev_ctx.author);
 }
 
-static void reset_dump_ctx(const char *url)
+static void reset_dump_ctx(const char *url, const char *dst_ref)
 {
 	strbuf_reset(&dump_ctx.url);
 	if (url)
@@ -90,6 +91,8 @@ static void reset_dump_ctx(const char *url)
 	dump_ctx.version = 1;
 	strbuf_reset(&dump_ctx.uuid);
 	dump_ctx.first_commit_done = 0;
+	strbuf_reset(&dump_ctx.ref_name);
+	strbuf_addstr(&dump_ctx.ref_name, dst_ref);
 }
 
 static void handle_property(const struct strbuf *key_buf,
@@ -336,8 +339,8 @@ static void begin_revision(void)
 
 	add_metadata_trailer(&rev_ctx.log);
 
-	fast_export_begin_commit("refs/heads/master", rev_ctx.revision, buf,
-		author, email.buf, &rev_ctx.log, rev_ctx.timestamp);
+	fast_export_begin_commit(dump_ctx.ref_name.buf, rev_ctx.revision, buf,
+				author, email.buf, &rev_ctx.log, rev_ctx.timestamp);
 }
 
 static void end_revision(void)
@@ -491,6 +494,9 @@ void svndump_read(void)
 
 int svndump_init(const struct svndump_options *o)
 {
+	const char *ref = o->ref;
+	if (!ref)
+		ref = "refs/heads/master";
 	if (buffer_init(&input, o->dumpfile))
 		return error("cannot open %s: %s", o->dumpfile, strerror(errno));
 	fast_export_init(REPORT_FILENO);
@@ -500,7 +506,7 @@ int svndump_init(const struct svndump_options *o)
 	strbuf_init(&rev_ctx.author, 4096);
 	strbuf_init(&node_ctx.src, 4096);
 	strbuf_init(&node_ctx.dst, 4096);
-	reset_dump_ctx(o->git_svn_url);
+	reset_dump_ctx(o->git_svn_url, ref);
 	reset_rev_ctx(0);
 	reset_node_ctx(NULL);
 	return 0;
@@ -509,7 +515,7 @@ int svndump_init(const struct svndump_options *o)
 void svndump_deinit(void)
 {
 	fast_export_deinit();
-	reset_dump_ctx(NULL);
+	reset_dump_ctx(NULL, "");
 	reset_rev_ctx(0);
 	reset_node_ctx(NULL);
 	strbuf_release(&rev_ctx.log);
diff --git a/vcs-svn/svndump.h b/vcs-svn/svndump.h
index db39dfe..0b01ccd 100644
--- a/vcs-svn/svndump.h
+++ b/vcs-svn/svndump.h
@@ -6,6 +6,7 @@ struct svndump_options {
 	 * dumpfile is opened in svndump_init and is read in svndump_read.
 	 */
 	const char *dumpfile, *git_svn_url;
+	const char *ref;
 };
 
 int svndump_init(const struct svndump_options *o);
-- 
1.7.3.4

[PATCH v3 07/10] vcs-svn,svn-fe: convert REPORT_FILENO to an option

[Dmitry Ivankov]

svn-fe needs to read fast-import's responses to "ls" and "cat-blob".
These come through a file descriptor number 3.

Descriptor number 3 isn't a standard of any kind and thus there is
no reason to hardcode it. Moreover it may require noticeable effort
to setup this descriptor, if number 3 is already taken for example.

Add an option for this file descriptor number in vcs-svn/ and svn-fe,
add a simple test for it.

To be used like following:
$ svn-fe --read-blob-fd=7 ... 7<somewhere

Signed-off-by: Dmitry Ivankov <divanorama@gmail.com>
---
 contrib/svn-fe/svn-fe.c   |    3 ++
 contrib/svn-fe/svn-fe.txt |    9 ++++++-
 t/t9010-svn-fe.sh         |   52 ++++++++++++++++++++++++++++++++++++++++++++-
 test-svn-fe.c             |    3 ++
 vcs-svn/svndump.c         |    4 +--
 vcs-svn/svndump.h         |    1 +
 6 files changed, 66 insertions(+), 6 deletions(-)

diff --git a/contrib/svn-fe/svn-fe.c b/contrib/svn-fe/svn-fe.c
index 9dd8336..a4fd7f5 100644
--- a/contrib/svn-fe/svn-fe.c
+++ b/contrib/svn-fe/svn-fe.c
@@ -19,11 +19,14 @@ static struct option svn_fe_options[] = {
 		"add git-svn-id line to log messages, imitating git-svn"),
 	OPT_STRING(0, "ref", &options.ref, "refname",
 		"write to <refname> instead of refs/heads/master"),
+	OPT_INTEGER(0, "read-blob-fd", &options.backflow_fd,
+		"read blobs and trees from this fd instead of 3"),
 	OPT_END()
 };
 
 int main(int argc, const char **argv)
 {
+	options.backflow_fd = 3;
 	argc = parse_options(argc, argv, NULL, svn_fe_options,
 						svn_fe_usage, 0);
 	if (argc > 1)
diff --git a/contrib/svn-fe/svn-fe.txt b/contrib/svn-fe/svn-fe.txt
index 0d19475..7bfc5a6 100644
--- a/contrib/svn-fe/svn-fe.txt
+++ b/contrib/svn-fe/svn-fe.txt
@@ -8,9 +8,9 @@ svn-fe - convert an SVN "dumpfile" to a fast-import stream
 SYNOPSIS
 --------
 [verse]
-mkfifo backchannel &&
+mkfifo backchannel && fd=3 &&
 svnadmin dump --deltas REPO |
-	svn-fe [options] [git-svn-id-url] 3<backchannel |
+	eval "svn-fe [options] [git-svn-id-url] $fd<backchannel" |
 	git fast-import --cat-blob-fd=3 3>backchannel
 
 DESCRIPTION
@@ -37,6 +37,11 @@ OPTIONS
 	Ref to be written by the generated stream.
 	Default is refs/heads/master.
 
+--read-blob-fd=<fd>::
+	Integer number of file descriptor from which
+	responses to 'ls' and 'cat-blob' requests will come.
+	Default is fd=3.
+
 INPUT FORMAT
 ------------
 Subversion's repository dump format is documented in full in
diff --git a/t/t9010-svn-fe.sh b/t/t9010-svn-fe.sh
index b45527e..efb4925 100755
--- a/t/t9010-svn-fe.sh
+++ b/t/t9010-svn-fe.sh
@@ -21,9 +21,10 @@ try_dump () {
 	maybe_fail_svnfe=${2:+test_$2} &&
 	maybe_fail_fi=${3:+test_$3} &&
 	args=${4:-} &&
+	fd=${5:-3} &&
 
 	{
-		$maybe_fail_svnfe test-svn-fe $args "$input" >stream 3<backflow &
+		eval "$maybe_fail_svnfe test-svn-fe $args "$input" >stream $fd<backflow" &
 	} &&
 	$maybe_fail_fi git fast-import --cat-blob-fd=3 <stream 3>backflow &&
 	wait $!
@@ -1121,4 +1122,53 @@ test_expect_success SVNREPO,PIPE 't9135/svn.dump' '
 	)
 '
 
+test_expect_success PIPE 'use different backflow fd' '
+	reinit_git &&
+	echo hi >hi &&
+	{
+		properties \
+			svn:author author@example.com \
+			svn:date "1999-02-01T00:01:002.000000Z" \
+			svn:log "add directory with some files in it" &&
+		echo PROPS-END
+	} >props &&
+	{
+		echo Prop-content-length: $(wc -c <props) &&
+		echo Content-length: $(wc -c <props) &&
+		echo &&
+		cat props
+	} >revprops &&
+	{
+		cat <<-EOF &&
+		SVN-fs-dump-format-version: 3
+
+		Revision-number: 1
+		EOF
+		cat revprops &&
+		cat <<-EOF &&
+		Node-path: directory
+		Node-kind: dir
+		Node-action: add
+		Node-path: directory/somefile
+		Node-kind: file
+		Node-action: add
+		EOF
+		text_no_props hi &&
+
+		echo "Revision-number: 2" &&
+		cat revprops &&
+		cat <<-\EOF
+		Node-path: otherfile
+		Node-kind: file
+		Node-action: add
+		Node-copyfrom-rev: 1
+		Node-copyfrom-path: directory/somefile
+		EOF
+	} >directory.dump &&
+	try_dump directory.dump "" "" "--read-blob-fd=7" 7 &&
+
+	git checkout HEAD otherfile &&
+	test_cmp hi otherfile
+'
+
 test_done
diff --git a/test-svn-fe.c b/test-svn-fe.c
index fddd3e8..568e47e 100644
--- a/test-svn-fe.c
+++ b/test-svn-fe.c
@@ -23,6 +23,8 @@ static struct option test_svnfe_options[] = {
 	OPT_SET_INT('d', "apply-delta", &delta_test, "test apply_delta", 1),
 	OPT_STRING(0, "ref", &options.ref, "refname",
 		"write to <refname> instead of refs/heads/master"),
+	OPT_INTEGER(0, "read-blob-fd", &options.backflow_fd,
+		"read blobs and trees from this fd instead of 3"),
 	OPT_END()
 };
 
@@ -54,6 +56,7 @@ static int apply_delta(int argc, const char *argv[])
 
 int main(int argc, const char *argv[])
 {
+	options.backflow_fd = 3;
 	argc = parse_options(argc, argv, NULL, test_svnfe_options,
 						test_svnfe_usage, 0);
 
diff --git a/vcs-svn/svndump.c b/vcs-svn/svndump.c
index 3a64708..847d733 100644
--- a/vcs-svn/svndump.c
+++ b/vcs-svn/svndump.c
@@ -20,8 +20,6 @@
  */
 #define constcmp(s, ref) memcmp(s, ref, sizeof(ref) - 1)
 
-#define REPORT_FILENO 3
-
 #define NODEACT_REPLACE 4
 #define NODEACT_DELETE 3
 #define NODEACT_ADD 2
@@ -499,7 +497,7 @@ int svndump_init(const struct svndump_options *o)
 		ref = "refs/heads/master";
 	if (buffer_init(&input, o->dumpfile))
 		return error("cannot open %s: %s", o->dumpfile, strerror(errno));
-	fast_export_init(REPORT_FILENO);
+	fast_export_init(o->backflow_fd);
 	strbuf_init(&dump_ctx.uuid, 4096);
 	strbuf_init(&dump_ctx.url, 4096);
 	strbuf_init(&rev_ctx.log, 4096);
diff --git a/vcs-svn/svndump.h b/vcs-svn/svndump.h
index 0b01ccd..8de9999 100644
--- a/vcs-svn/svndump.h
+++ b/vcs-svn/svndump.h
@@ -7,6 +7,7 @@ struct svndump_options {
 	 */
 	const char *dumpfile, *git_svn_url;
 	const char *ref;
+	int backflow_fd;
 };
 
 int svndump_init(const struct svndump_options *o);
-- 
1.7.3.4

[PATCH v3 08/10] vcs-svn,svn-fe: allow to disable 'progress' lines

[Dmitry Ivankov]

vcs-svn/ writes a progress line after each processed revision. It
is too noisy for big imports. That's a stress for a terminal and
any other output can be lost or scrolled away among these lines.
If svn-fe is invoked by a remote helper the import stream with
progress lines in it will go directly to the git fast-import which
always prints every progress line met in the stream.

For now just add a switch to turn progress lines off:
$ svn-fe --no-progress ...

Signed-off-by: Dmitry Ivankov <divanorama@gmail.com>
---
 contrib/svn-fe/svn-fe.c   |    3 +++
 contrib/svn-fe/svn-fe.txt |    4 ++++
 test-svn-fe.c             |    1 +
 vcs-svn/svndump.c         |    6 +++++-
 vcs-svn/svndump.h         |    1 +
 5 files changed, 14 insertions(+), 1 deletions(-)

diff --git a/contrib/svn-fe/svn-fe.c b/contrib/svn-fe/svn-fe.c
index a4fd7f5..6aedba2 100644
--- a/contrib/svn-fe/svn-fe.c
+++ b/contrib/svn-fe/svn-fe.c
@@ -15,6 +15,9 @@ static const char * const svn_fe_usage[] = {
 static struct svndump_options options;
 
 static struct option svn_fe_options[] = {
+	{ OPTION_SET_INT, 0, "progress", &options.progress,
+		NULL, "don't write a progress line after each commit",
+		PARSE_OPT_NOARG | PARSE_OPT_NEGHELP, NULL, 1 },
 	OPT_STRING(0, "git-svn-id-url", &options.git_svn_url, "url",
 		"add git-svn-id line to log messages, imitating git-svn"),
 	OPT_STRING(0, "ref", &options.ref, "refname",
diff --git a/contrib/svn-fe/svn-fe.txt b/contrib/svn-fe/svn-fe.txt
index 7bfc5a6..abeebfe 100644
--- a/contrib/svn-fe/svn-fe.txt
+++ b/contrib/svn-fe/svn-fe.txt
@@ -42,6 +42,10 @@ OPTIONS
 	responses to 'ls' and 'cat-blob' requests will come.
 	Default is fd=3.
 
+--[no-]progress::
+	Write 'progress' lines to fast-import stream. These
+	can be displayed by fast-import.
+
 INPUT FORMAT
 ------------
 Subversion's repository dump format is documented in full in
diff --git a/test-svn-fe.c b/test-svn-fe.c
index 568e47e..e51a9bb 100644
--- a/test-svn-fe.c
+++ b/test-svn-fe.c
@@ -57,6 +57,7 @@ static int apply_delta(int argc, const char *argv[])
 int main(int argc, const char *argv[])
 {
 	options.backflow_fd = 3;
+	options.progress = 1;
 	argc = parse_options(argc, argv, NULL, test_svnfe_options,
 						test_svnfe_usage, 0);
 
diff --git a/vcs-svn/svndump.c b/vcs-svn/svndump.c
index 847d733..ec68649 100644
--- a/vcs-svn/svndump.c
+++ b/vcs-svn/svndump.c
@@ -37,6 +37,8 @@
 
 #define MAX_GITSVN_LINE_LEN 4096
 
+static int print_progress;
+
 static struct line_buffer input = LINE_BUFFER_INIT;
 
 static struct {
@@ -344,7 +346,8 @@ static void begin_revision(void)
 static void end_revision(void)
 {
 	if (rev_ctx.revision) {
-		fast_export_progress(rev_ctx.revision);
+		if (print_progress)
+			fast_export_progress(rev_ctx.revision);
 		dump_ctx.first_commit_done = 1;
 	}
 }
@@ -497,6 +500,7 @@ int svndump_init(const struct svndump_options *o)
 		ref = "refs/heads/master";
 	if (buffer_init(&input, o->dumpfile))
 		return error("cannot open %s: %s", o->dumpfile, strerror(errno));
+	print_progress = o->progress;
 	fast_export_init(o->backflow_fd);
 	strbuf_init(&dump_ctx.uuid, 4096);
 	strbuf_init(&dump_ctx.url, 4096);
diff --git a/vcs-svn/svndump.h b/vcs-svn/svndump.h
index 8de9999..de54969 100644
--- a/vcs-svn/svndump.h
+++ b/vcs-svn/svndump.h
@@ -8,6 +8,7 @@ struct svndump_options {
 	const char *dumpfile, *git_svn_url;
 	const char *ref;
 	int backflow_fd;
+	int progress;
 };
 
 int svndump_init(const struct svndump_options *o);
-- 
1.7.3.4

[PATCH v3 09/10] vcs-svn,svn-fe: add --incremental option

[Dmitry Ivankov]

This option is to make svn-fe write commits on top of the existing ref
instead of overwriting it. More precise, the first commit's parent is
set to be :(first_revision_in_current_dump - 1) mark.

Prerequisite is to (re)use import marks (from previous imports). It is
safe to use this option on a svn dump that starts with r0/r1. The svn
dump itself should be incremental too.

Signed-off-by: Dmitry Ivankov <divanorama@gmail.com>
---
 contrib/svn-fe/svn-fe.c   |    3 +++
 contrib/svn-fe/svn-fe.txt |    6 ++++++
 t/t9010-svn-fe.sh         |   26 +++++++++++++++++++++++++-
 test-svn-fe.c             |    3 +++
 vcs-svn/svndump.c         |   13 +++++++++----
 vcs-svn/svndump.h         |    2 +-
 6 files changed, 47 insertions(+), 6 deletions(-)

diff --git a/contrib/svn-fe/svn-fe.c b/contrib/svn-fe/svn-fe.c
index 6aedba2..9e0be17 100644
--- a/contrib/svn-fe/svn-fe.c
+++ b/contrib/svn-fe/svn-fe.c
@@ -18,6 +18,9 @@ static struct option svn_fe_options[] = {
 	{ OPTION_SET_INT, 0, "progress", &options.progress,
 		NULL, "don't write a progress line after each commit",
 		PARSE_OPT_NOARG | PARSE_OPT_NEGHELP, NULL, 1 },
+	OPT_SET_INT(0, "incremental", &options.incremental,
+		"resume export, requires marks and incremental dump",
+		1),
 	OPT_STRING(0, "git-svn-id-url", &options.git_svn_url, "url",
 		"add git-svn-id line to log messages, imitating git-svn"),
 	OPT_STRING(0, "ref", &options.ref, "refname",
diff --git a/contrib/svn-fe/svn-fe.txt b/contrib/svn-fe/svn-fe.txt
index abeebfe..f13648c 100644
--- a/contrib/svn-fe/svn-fe.txt
+++ b/contrib/svn-fe/svn-fe.txt
@@ -46,6 +46,12 @@ OPTIONS
 	Write 'progress' lines to fast-import stream. These
 	can be displayed by fast-import.
 
+--incremental::
+	If the first revision in dump has number greater than
+	1, make :(revision - 1) it's parent. For this to work
+	fast-import must be supplied with import-marks file
+	and the dump must be incremental.
+
 INPUT FORMAT
 ------------
 Subversion's repository dump format is documented in full in
diff --git a/t/t9010-svn-fe.sh b/t/t9010-svn-fe.sh
index efb4925..845e227 100755
--- a/t/t9010-svn-fe.sh
+++ b/t/t9010-svn-fe.sh
@@ -22,11 +22,12 @@ try_dump () {
 	maybe_fail_fi=${3:+test_$3} &&
 	args=${4:-} &&
 	fd=${5:-3} &&
+	fi_args=${6:-} &&
 
 	{
 		eval "$maybe_fail_svnfe test-svn-fe $args "$input" >stream $fd<backflow" &
 	} &&
-	$maybe_fail_fi git fast-import --cat-blob-fd=3 <stream 3>backflow &&
+	eval "$maybe_fail_fi git fast-import $fi_args --cat-blob-fd=3 <stream 3>backflow" &&
 	wait $!
 }
 
@@ -71,6 +72,15 @@ Content-length: 10
 
 PROPS-END
 EOF
+cat >moreempty.dump <<-EOF &&
+SVN-fs-dump-format-version: 3
+
+Revision-number: 3
+Prop-content-length: 10
+Content-length: 10
+
+PROPS-END
+EOF
 
 test_expect_success 'setup: have pipes?' '
 	rm -f frob &&
@@ -1171,4 +1181,18 @@ test_expect_success PIPE 'use different backflow fd' '
 	test_cmp hi otherfile
 '
 
+test_expect_success PIPE 'incremental import' '
+	reinit_git &&
+	>./marks &&
+
+	try_dump emptyprop.dump "" "" "--incremental" "" "--export-marks=./marks" &&
+	test_line_count = 2 ./marks &&
+
+	try_dump moreempty.dump "" "" "--incremental" "" "--import-marks=./marks --export-marks=./marks" &&
+	test_line_count = 3 ./marks &&
+
+	git log --format=oneline >history &&
+	test_line_count = 3 ./history
+'
+
 test_done
diff --git a/test-svn-fe.c b/test-svn-fe.c
index e51a9bb..206827e 100644
--- a/test-svn-fe.c
+++ b/test-svn-fe.c
@@ -23,6 +23,9 @@ static struct option test_svnfe_options[] = {
 	OPT_SET_INT('d', "apply-delta", &delta_test, "test apply_delta", 1),
 	OPT_STRING(0, "ref", &options.ref, "refname",
 		"write to <refname> instead of refs/heads/master"),
+	OPT_SET_INT(0, "incremental", &options.incremental,
+		"resume export, requires marks and incremental dump",
+		1),
 	OPT_INTEGER(0, "read-blob-fd", &options.backflow_fd,
 		"read blobs and trees from this fd instead of 3"),
 	OPT_END()
diff --git a/vcs-svn/svndump.c b/vcs-svn/svndump.c
index ec68649..75ea49f 100644
--- a/vcs-svn/svndump.c
+++ b/vcs-svn/svndump.c
@@ -58,6 +58,7 @@ static struct {
 	struct strbuf uuid, url;
 	int first_commit_done;
 	struct strbuf ref_name;
+	int incremental;
 } dump_ctx;
 
 static void reset_node_ctx(char *fname)
@@ -83,7 +84,7 @@ static void reset_rev_ctx(uint32_t revision)
 	strbuf_reset(&rev_ctx.author);
 }
 
-static void reset_dump_ctx(const char *url, const char *dst_ref)
+static void reset_dump_ctx(const char *url, const char *dst_ref, int incremental)
 {
 	strbuf_reset(&dump_ctx.url);
 	if (url)
@@ -93,6 +94,7 @@ static void reset_dump_ctx(const char *url, const char *dst_ref)
 	dump_ctx.first_commit_done = 0;
 	strbuf_reset(&dump_ctx.ref_name);
 	strbuf_addstr(&dump_ctx.ref_name, dst_ref);
+	dump_ctx.incremental = incremental;
 }
 
 static void handle_property(const struct strbuf *key_buf,
@@ -322,7 +324,10 @@ static void begin_revision(void)
 
 	if (!rev_ctx.revision)	/* revision 0 gets no git commit. */
 		return;
-	prev = dump_ctx.first_commit_done ? rev_ctx.revision - 1 : 0;
+	if (dump_ctx.incremental)
+		prev = rev_ctx.revision - 1;
+	else
+		prev = dump_ctx.first_commit_done ? rev_ctx.revision - 1 : 0;
 	if (prev)
 		snprintf(buf, 32, ":%"PRIu32, prev);
 	else
@@ -508,7 +513,7 @@ int svndump_init(const struct svndump_options *o)
 	strbuf_init(&rev_ctx.author, 4096);
 	strbuf_init(&node_ctx.src, 4096);
 	strbuf_init(&node_ctx.dst, 4096);
-	reset_dump_ctx(o->git_svn_url, ref);
+	reset_dump_ctx(o->git_svn_url, ref, o->incremental);
 	reset_rev_ctx(0);
 	reset_node_ctx(NULL);
 	return 0;
@@ -517,7 +522,7 @@ int svndump_init(const struct svndump_options *o)
 void svndump_deinit(void)
 {
 	fast_export_deinit();
-	reset_dump_ctx(NULL, "");
+	reset_dump_ctx(NULL, "", 0);
 	reset_rev_ctx(0);
 	reset_node_ctx(NULL);
 	strbuf_release(&rev_ctx.log);
diff --git a/vcs-svn/svndump.h b/vcs-svn/svndump.h
index de54969..32062e4 100644
--- a/vcs-svn/svndump.h
+++ b/vcs-svn/svndump.h
@@ -8,7 +8,7 @@ struct svndump_options {
 	const char *dumpfile, *git_svn_url;
 	const char *ref;
 	int backflow_fd;
-	int progress;
+	int progress, incremental;
 };
 
 int svndump_init(const struct svndump_options *o);
-- 
1.7.3.4

[PATCH v3 10/10] vcs-svn: add fast_export_note to create notes

[Dmitry Ivankov]

fast_export lacked a method to writes notes to fast-import stream.
Add two new functions fast_export_note which is similar to
fast_export_modify. And also add fast_export_buf_to_data to be able
to write inline blobs that don't come from a line_buffer or from delta
application.

To be used like this:
fast_export_begin_commit("refs/notes/somenotes", ...)

fast_export_note("refs/heads/master", "inline")
fast_export_buf_to_data(&data)
or maybe
fast_export_note("refs/heads/master", sha1)

Signed-off-by: Dmitry Ivankov <divanorama@gmail.com>
---
 vcs-svn/fast_export.c |   11 +++++++++++
 vcs-svn/fast_export.h |    2 ++
 2 files changed, 13 insertions(+), 0 deletions(-)

diff --git a/vcs-svn/fast_export.c b/vcs-svn/fast_export.c
index 3dfccd2..07220b0 100644
--- a/vcs-svn/fast_export.c
+++ b/vcs-svn/fast_export.c
@@ -69,6 +69,10 @@ void fast_export_modify(const char *path, uint32_t mode, const char *dataref)
 	putchar('\n');
 }
 
+void fast_export_note(const char *committish, const char *dataref) {
+	printf("N %s %s\n", dataref, committish);
+}
+
 void fast_export_begin_commit(const char *ref, uint32_t mark, const char *from,
 			const char *author_name, const char *author_email,
 			const struct strbuf *log, unsigned long timestamp)
@@ -206,6 +210,13 @@ static long apply_delta(off_t len, struct line_buffer *input,
 	return ret;
 }
 
+void fast_export_buf_to_data(const struct strbuf *data)
+{
+	printf("data %"PRIuMAX"\n", (uintmax_t)data->len);
+	fwrite(data->buf, data->len, 1, stdout);
+	fputc('\n', stdout);
+}
+
 void fast_export_data(uint32_t mode, uint32_t len, struct line_buffer *input)
 {
 	if (mode == REPO_MODE_LNK) {
diff --git a/vcs-svn/fast_export.h b/vcs-svn/fast_export.h
index bf58880..d17e5d3 100644
--- a/vcs-svn/fast_export.h
+++ b/vcs-svn/fast_export.h
@@ -10,10 +10,12 @@ void fast_export_reset(void);
 
 void fast_export_delete(const char *path);
 void fast_export_modify(const char *path, uint32_t mode, const char *dataref);
+void fast_export_note(const char *committish, const char *dataref);
 void fast_export_begin_commit(const char *ref, uint32_t mark, const char *from,
 			const char *author_name, const char *author_email,
 			const struct strbuf *log, unsigned long timestamp);
 void fast_export_progress(uint32_t revision);
+void fast_export_buf_to_data(const struct strbuf *data);
 void fast_export_data(uint32_t mode, uint32_t len, struct line_buffer *input);
 void fast_export_blob_delta(uint32_t mode,
 			uint32_t old_mode, const char *old_data,
-- 
1.7.3.4