[RFCv2 00/12] Foreign VCS helper program for CVS repositories

[RFCv2 00/12] Foreign VCS helper program for CVS repositories

[Johan Herland]

Hi,

Here is a resend of the jh/vcs-cvs topic in 'pu'. This series is based on
the jh/notes topic in current 'pu', but is NOT based on the db/transport-shim
topic. As a consequence of this, the series now contains 3 more patches that
were part of the original db/foreign-scm topic.

Also, I expect that Barkalow will at some point send an updated
db/foreign-scm topic, which will invalidate parts of - or the whole of -
this patch series. I expect to rebase and resend this topic when the
updated db/foreign-scm appears, but since I am away for the next week,
I still send this series now, to reflect the current (unfinished) state.

The 7 first patches are a resend of Barkalow's original foreign-scm topic,
or at least enough of it to support the rest of this series.

Next, there are 3 patches tweaking and expanding the foreign-vcs API (with
corresponding implementations in the foreign transport code) to adjust for
the CVS helper's needs.

The final 2 patches add the git-vcs-cvs helper program that implements the
fetch/import of objects from a local or remote CVS repository, along with
selftests for verifying its correctness in some simple cases.


Changes since the last iteration:
- Added 3 more patches from the db/foreign-scm topic, caused by rebase.
- The patch with preliminary refactoring in fast-import.c is gone.
- The patch teaching fast-import to import notes has been moved to jh/notes.
- The CVS helper is updated to use the new 'notemodify' fast-import command.


Daniel Barkalow (7):
  Allow late reporting of fetched hashes
  Document details of transport function APIs
  Add option for using a foreign VCS
  Add specification of git-vcs-* helper programs
  Use a function to determine whether a remote is valid
  Allow programs to not depend on remotes having urls
  Add a transport implementation using git-vcs-* helpers

Johan Herland (5):
  Preliminary clarifications to git-vcs documentation
  Teach foreign transport code to perform the "capabilities" command
  Introduce a 'marks <filename>' feature to the foreign transport code
  First draft of CVS importer using the foreign-scm machinery
  Add simple test cases of git-vcs-cvs functionality

 Documentation/config.txt         |    4 +
 Documentation/git-vcs-cvs.txt    |   85 ++++
 Documentation/git-vcs.txt        |   88 ++++
 Makefile                         |   47 ++
 builtin-clone.c                  |    6 +-
 builtin-fetch.c                  |   19 +-
 builtin-ls-remote.c              |    6 +-
 builtin-push.c                   |   54 ++-
 configure.ac                     |    3 +
 git-vcs-cvs.py                   |  697 ++++++++++++++++++++++++++++++
 git_vcs_cvs/.gitignore           |    2 +
 git_vcs_cvs/Makefile             |   27 ++
 git_vcs_cvs/changeset.py         |  114 +++++
 git_vcs_cvs/commit_states.py     |   52 +++
 git_vcs_cvs/cvs.py               |  884 ++++++++++++++++++++++++++++++++++++++
 git_vcs_cvs/cvs_revision_map.py  |  367 ++++++++++++++++
 git_vcs_cvs/cvs_symbol_cache.py  |  283 ++++++++++++
 git_vcs_cvs/git.py               |  591 +++++++++++++++++++++++++
 git_vcs_cvs/setup.py             |   12 +
 git_vcs_cvs/util.py              |  147 +++++++
 remote.c                         |   15 +-
 remote.h                         |    2 +
 t/t9800-foreign-vcs-cvs-basic.sh |  518 ++++++++++++++++++++++
 t/t9801-foreign-vcs-cvs-fetch.sh |  291 +++++++++++++
 t/test-lib.sh                    |    1 +
 transport-foreign.c              |  271 ++++++++++++
 transport.c                      |   24 +-
 transport.h                      |   45 ++-
 28 files changed, 4609 insertions(+), 46 deletions(-)
 create mode 100644 Documentation/git-vcs-cvs.txt
 create mode 100644 Documentation/git-vcs.txt
 create mode 100755 git-vcs-cvs.py
 create mode 100644 git_vcs_cvs/.gitignore
 create mode 100644 git_vcs_cvs/Makefile
 create mode 100644 git_vcs_cvs/__init__.py
 create mode 100644 git_vcs_cvs/changeset.py
 create mode 100644 git_vcs_cvs/commit_states.py
 create mode 100644 git_vcs_cvs/cvs.py
 create mode 100644 git_vcs_cvs/cvs_revision_map.py
 create mode 100644 git_vcs_cvs/cvs_symbol_cache.py
 create mode 100644 git_vcs_cvs/git.py
 create mode 100644 git_vcs_cvs/setup.py
 create mode 100644 git_vcs_cvs/util.py
 create mode 100755 t/t9800-foreign-vcs-cvs-basic.sh
 create mode 100755 t/t9801-foreign-vcs-cvs-fetch.sh
 create mode 100644 transport-foreign.c


Have fun! :)

...Johan

[RFCv2 01/12] Allow late reporting of fetched hashes

[Johan Herland]

From: Daniel Barkalow <barkalow@iabervon.org>

Some future transports (in particular, foreign VCS importers) will
only report the hashes of new commits when the objects are also
available. In preparation, allow fetch_refs() to modify the refs it
gets (in particular, the remote side's sha1), and treat the null sha1,
when reported by get_ref_list(), as different from any value,
including itself (which, when local, indicates that the local version
doesn't exist yet).

Signed-off-by: Daniel Barkalow <barkalow@iabervon.org>
Signed-off-by: Johan Herland <johan@herland.net>
---
 builtin-clone.c |    6 ++++--
 transport.c     |   17 +++++++++--------
 transport.h     |    4 ++--
 3 files changed, 15 insertions(+), 12 deletions(-)

diff --git a/builtin-clone.c b/builtin-clone.c
index 32dea74..f281756 100644
--- a/builtin-clone.c
+++ b/builtin-clone.c
@@ -509,8 +509,10 @@ int cmd_clone(int argc, const char **argv, const char *prefix)
 					     option_upload_pack);
 
 		refs = transport_get_remote_refs(transport);
-		if(refs)
-			transport_fetch_refs(transport, refs);
+		if (refs) {
+			struct ref *ref_cpy = copy_ref_list(refs);
+			transport_fetch_refs(transport, ref_cpy);
+		}
 	}
 
 	if (refs) {
diff --git a/transport.c b/transport.c
index 8a42e76..349ccae 100644
--- a/transport.c
+++ b/transport.c
@@ -207,7 +207,7 @@ static struct ref *get_refs_via_rsync(struct transport *transport, int for_push)
 }
 
 static int fetch_objs_via_rsync(struct transport *transport,
-				int nr_objs, const struct ref **to_fetch)
+				int nr_objs, struct ref **to_fetch)
 {
 	struct strbuf buf = STRBUF_INIT;
 	struct child_process rsync;
@@ -356,7 +356,7 @@ static int rsync_transport_push(struct transport *transport,
 
 #ifndef NO_CURL /* http fetch is the only user */
 static int fetch_objs_via_walker(struct transport *transport,
-				 int nr_objs, const struct ref **to_fetch)
+				 int nr_objs, struct ref **to_fetch)
 {
 	char *dest = xstrdup(transport->url);
 	struct walker *walker = transport->data;
@@ -500,7 +500,7 @@ static struct ref *get_refs_via_curl(struct transport *transport, int for_push)
 }
 
 static int fetch_objs_via_curl(struct transport *transport,
-				 int nr_objs, const struct ref **to_fetch)
+				 int nr_objs, struct ref **to_fetch)
 {
 	if (!transport->data)
 		transport->data = get_http_walker(transport->url,
@@ -540,7 +540,7 @@ static struct ref *get_refs_from_bundle(struct transport *transport, int for_pus
 }
 
 static int fetch_refs_from_bundle(struct transport *transport,
-			       int nr_heads, const struct ref **to_fetch)
+			       int nr_heads, struct ref **to_fetch)
 {
 	struct bundle_transport_data *data = transport->data;
 	return unbundle(&data->header, data->fd);
@@ -618,7 +618,7 @@ static struct ref *get_refs_via_connect(struct transport *transport, int for_pus
 }
 
 static int fetch_refs_via_pack(struct transport *transport,
-			       int nr_heads, const struct ref **to_fetch)
+			       int nr_heads, struct ref **to_fetch)
 {
 	struct git_transport_data *data = transport->data;
 	char **heads = xmalloc(nr_heads * sizeof(*heads));
@@ -1032,15 +1032,16 @@ const struct ref *transport_get_remote_refs(struct transport *transport)
 	return transport->remote_refs;
 }
 
-int transport_fetch_refs(struct transport *transport, const struct ref *refs)
+int transport_fetch_refs(struct transport *transport, struct ref *refs)
 {
 	int rc;
 	int nr_heads = 0, nr_alloc = 0;
-	const struct ref **heads = NULL;
-	const struct ref *rm;
+	struct ref **heads = NULL;
+	struct ref *rm;
 
 	for (rm = refs; rm; rm = rm->next) {
 		if (rm->peer_ref &&
+		    !is_null_sha1(rm->old_sha1) &&
 		    !hashcmp(rm->peer_ref->old_sha1, rm->old_sha1))
 			continue;
 		ALLOC_GROW(heads, nr_heads + 1, nr_alloc);
diff --git a/transport.h b/transport.h
index 51b5397..3cb0abc 100644
--- a/transport.h
+++ b/transport.h
@@ -19,7 +19,7 @@ struct transport {
 			  const char *value);
 
 	struct ref *(*get_refs_list)(struct transport *transport, int for_push);
-	int (*fetch)(struct transport *transport, int refs_nr, const struct ref **refs);
+	int (*fetch)(struct transport *transport, int refs_nr, struct ref **refs);
 	int (*push_refs)(struct transport *transport, struct ref *refs, int flags);
 	int (*push)(struct transport *connection, int refspec_nr, const char **refspec, int flags);
 
@@ -72,7 +72,7 @@ int transport_push(struct transport *connection,
 
 const struct ref *transport_get_remote_refs(struct transport *transport);
 
-int transport_fetch_refs(struct transport *transport, const struct ref *refs);
+int transport_fetch_refs(struct transport *transport, struct ref *refs);
 void transport_unlock_pack(struct transport *transport);
 int transport_disconnect(struct transport *transport);
 char *transport_anonymize_url(const char *url);
-- 
1.6.4.rc3.138.ga6b98.dirty

[RFCv2 02/12] Document details of transport function APIs

[Johan Herland]

From: Daniel Barkalow <barkalow@iabervon.org>

In particular, explain which of the fields of struct ref is used for
what purpose in the input to and output from each function.

Signed-off-by: Daniel Barkalow <barkalow@iabervon.org>
Signed-off-by: Johan Herland <johan@herland.net>
---
 transport.h |   38 ++++++++++++++++++++++++++++++++++++++
 1 files changed, 38 insertions(+), 0 deletions(-)

diff --git a/transport.h b/transport.h
index 3cb0abc..b45e6c5 100644
--- a/transport.h
+++ b/transport.h
@@ -18,11 +18,49 @@ struct transport {
 	int (*set_option)(struct transport *connection, const char *name,
 			  const char *value);
 
+	/**
+	 * Returns a list of the remote side's refs. In order to allow
+	 * the transport to try to share connections, for_push is a
+	 * hint as to whether the ultimate operation is a push or a fetch.
+	 *
+	 * If the transport is able to determine the remote hash for
+	 * the ref without a huge amount of effort, it should store it
+	 * in the ref's old_sha1 field; otherwise it should be all 0.
+	 **/
 	struct ref *(*get_refs_list)(struct transport *transport, int for_push);
+
+	/**
+	 * Fetch the objects for the given refs. Note that this gets
+	 * an array, and should ignore the list structure.
+	 *
+	 * If the transport did not get hashes for refs in
+	 * get_refs_list(), it should set the old_sha1 fields in the
+	 * provided refs now.
+	 **/
 	int (*fetch)(struct transport *transport, int refs_nr, struct ref **refs);
+
+	/**
+	 * Push the objects and refs. Send the necessary objects, and
+	 * then tell the remote side to update each ref in the list
+	 * from old_sha1 to new_sha1.
+	 *
+	 * Where possible, set the status for each ref appropriately.
+	 *
+	 * If, in the process, the transport determines that the
+	 * remote side actually responded to the push by updating the
+	 * ref to a different value, the transport should modify the
+	 * new_sha1 in the ref. (Note that this is a matter of the
+	 * remote accepting but rewriting the change, not rejecting it
+	 * and reporting that a different update had already taken
+	 * place)
+	 **/
 	int (*push_refs)(struct transport *transport, struct ref *refs, int flags);
 	int (*push)(struct transport *connection, int refspec_nr, const char **refspec, int flags);
 
+	/** get_refs_list(), fetch(), and push_refs() can keep
+	 * resources (such as a connection) reserved for futher
+	 * use. disconnect() releases these resources.
+	 **/
 	int (*disconnect)(struct transport *connection);
 	char *pack_lockfile;
 	signed verbose : 2;
-- 
1.6.4.rc3.138.ga6b98.dirty

[RFCv2 03/12] Add option for using a foreign VCS

[Johan Herland]

From: Daniel Barkalow <barkalow@iabervon.org>

This simply configures the remote to use a transport that doesn't have
any methods at all and is therefore unable to do anything yet.

Signed-off-by: Daniel Barkalow <barkalow@iabervon.org>
Signed-off-by: Johan Herland <johan@herland.net>
---
 Documentation/config.txt |    4 ++++
 remote.c                 |    2 ++
 remote.h                 |    2 ++
 transport.c              |    3 ++-
 4 files changed, 10 insertions(+), 1 deletions(-)

diff --git a/Documentation/config.txt b/Documentation/config.txt
index 1446007..b4ab67b 100644
--- a/Documentation/config.txt
+++ b/Documentation/config.txt
@@ -1377,6 +1377,10 @@ remote.<name>.tagopt::
 	Setting this value to \--no-tags disables automatic tag following when
 	fetching from remote <name>
 
+remote.<name>.vcs::
+	Setting this to a value <vcs> will cause git to interact with
+	the remote with the git-vcs-<vcs> helper.
+
 remotes.<group>::
 	The list of remotes which are fetched by "git remote update
 	<group>".  See linkgit:git-remote[1].
diff --git a/remote.c b/remote.c
index c3ada2d..758727c 100644
--- a/remote.c
+++ b/remote.c
@@ -422,6 +422,8 @@ static int handle_config(const char *key, const char *value, void *cb)
 	} else if (!strcmp(subkey, ".proxy")) {
 		return git_config_string((const char **)&remote->http_proxy,
 					 key, value);
+	} else if (!strcmp(subkey, ".vcs")) {
+		return git_config_string(&remote->foreign_vcs, key, value);
 	}
 	return 0;
 }
diff --git a/remote.h b/remote.h
index 5db8420..ac0ce2f 100644
--- a/remote.h
+++ b/remote.h
@@ -11,6 +11,8 @@ struct remote {
 	const char *name;
 	int origin;
 
+	const char *foreign_vcs;
+
 	const char **url;
 	int url_nr;
 	int url_alloc;
diff --git a/transport.c b/transport.c
index 349ccae..f414bd3 100644
--- a/transport.c
+++ b/transport.c
@@ -928,7 +928,8 @@ struct transport *transport_get(struct remote *remote, const char *url)
 	ret->remote = remote;
 	ret->url = url;
 
-	if (!prefixcmp(url, "rsync:")) {
+	if (remote && remote->foreign_vcs) {
+	} else if (!prefixcmp(url, "rsync:")) {
 		ret->get_refs_list = get_refs_via_rsync;
 		ret->fetch = fetch_objs_via_rsync;
 		ret->push = rsync_transport_push;
-- 
1.6.4.rc3.138.ga6b98.dirty

[RFCv2 04/12] Add specification of git-vcs-* helper programs

[Johan Herland]

From: Daniel Barkalow <barkalow@iabervon.org>

Signed-off-by: Daniel Barkalow <barkalow@iabervon.org>
Signed-off-by: Johan Herland <johan@herland.net>
---
 Documentation/git-vcs.txt |   77 +++++++++++++++++++++++++++++++++++++++++++++
 1 files changed, 77 insertions(+), 0 deletions(-)
 create mode 100644 Documentation/git-vcs.txt

diff --git a/Documentation/git-vcs.txt b/Documentation/git-vcs.txt
new file mode 100644
index 0000000..402c927
--- /dev/null
+++ b/Documentation/git-vcs.txt
@@ -0,0 +1,77 @@
+git-vcs(1)
+============
+
+NAME
+----
+git-vcs - Helper programs for interoperation with foreign systems
+
+SYNOPSIS
+--------
+'git vcs-<system>' <remote>
+
+DESCRIPTION
+-----------
+
+These programs are normally not used directly by end users, but are
+invoked by various git programs that interact with remote repositories
+when the repository they would operate on uses a foreign system.
+
+Each 'git vcs-<system>' is a helper for interoperating with a
+particular version control system. Different helpers have different
+capabilities (limited both by the particular helper and by the
+capabilities of the system they connect to), and they report what
+capabilities they support.
+
+These programs can store refs in refs/<system>/*, and arbitrary
+information in info/<system>.
+
+COMMANDS
+--------
+
+Commands are given by the caller on the helper's standard input, one per line.
+
+'capabilities'::
+	Outputs a single line with a list of feature names separated
+	by spaces. Each of these indicates a supported feature of the
+	helper, and the caller will only attempt operations that are
+	supported.
+
+'list'::
+	Outputs the names of refs, one per line. These may be
+	followed, after a single space, by "changed" or "unchanged",
+	indicating whether the foreign repository has changed from the
+	state in the ref. If the helper doesn't know, it doesn't have
+	to provide a value. (In particular, it shouldn't do expensive
+	operations, such as importing the content, to see whether it
+	matches.) Other information, not yet supported, may be output
+	as well, separated by single spaces.
+
+'import' ref::
+	Imports the given ref by outputting it in git-fast-import
+	format.
+
+'export' commit ref::
+	Sends the given commit to the foreign system, with the
+	location given by ref, and reimports it by outputting it in
+	git-fast-import format as the foreign system rendered it.
++
+All parents of commit must either have been created with 'import' or
+have been passed to 'export' previously. Depending on the features,
+there may be other restrictions on what may be exported.
+
+FEATURES
+--------
+
+'export'::
+	Helper supports exporting commits, at least exporting
+	non-merge commits whose parents are not the parents of any
+	other commit exported to the same branch or make in the other
+	system on the same branch.
+
+'export-branch'::
+	Helper supports creating new branches by exporting commits to
+	them.
+
+'export-merges'::
+	Helper supports exporting two-parent merges, where both
+	parents have already been exported successfully.
-- 
1.6.4.rc3.138.ga6b98.dirty

[RFCv2 05/12] Use a function to determine whether a remote is valid

[Johan Herland]

From: Daniel Barkalow <barkalow@iabervon.org>

Currently, it only checks url, but it will allow other things in the future.

Signed-off-by: Daniel Barkalow <barkalow@iabervon.org>
Signed-off-by: Johan Herland <johan@herland.net>
---
 remote.c |   13 +++++++++----
 1 files changed, 9 insertions(+), 4 deletions(-)

diff --git a/remote.c b/remote.c
index 758727c..106151f 100644
--- a/remote.c
+++ b/remote.c
@@ -48,6 +48,11 @@ static int rewrite_nr;
 #define BUF_SIZE (2048)
 static char buffer[BUF_SIZE];
 
+static int valid_remote(const struct remote *remote)
+{
+	return !!remote->url;
+}
+
 static const char *alias_url(const char *url)
 {
 	int i, j;
@@ -669,14 +674,14 @@ struct remote *remote_get(const char *name)
 
 	ret = make_remote(name, 0);
 	if (valid_remote_nick(name)) {
-		if (!ret->url)
+		if (!valid_remote(ret))
 			read_remotes_file(ret);
-		if (!ret->url)
+		if (!valid_remote(ret))
 			read_branches_file(ret);
 	}
-	if (name_given && !ret->url)
+	if (name_given && !valid_remote(ret))
 		add_url_alias(ret, name);
-	if (!ret->url)
+	if (!valid_remote(ret))
 		return NULL;
 	ret->fetch = parse_fetch_refspec(ret->fetch_refspec_nr, ret->fetch_refspec);
 	ret->push = parse_push_refspec(ret->push_refspec_nr, ret->push_refspec);
-- 
1.6.4.rc3.138.ga6b98.dirty

[RFCv2 06/12] Allow programs to not depend on remotes having urls

[Johan Herland]

From: Daniel Barkalow <barkalow@iabervon.org>

For fetch and ls-remote, which use the first url of a remote, have
transport_get() determine this by passing a remote and passing NULL
for the url. For push, which uses every url of a remote, use each url
in turn if there are any, and use NULL if there are none.

This will allow the transport code to do something different if the
location is not specified with a url.

Also, have the message for a fetch say "foreign" if there is no url.

This patch has been improved by the following contributions:
- Alex Riesen: fix typo

Signed-off-by: Daniel Barkalow <barkalow@iabervon.org>
Signed-off-by: Johan Herland <johan@herland.net>
---
 builtin-fetch.c     |   19 +++++++++++------
 builtin-ls-remote.c |    6 ++--
 builtin-push.c      |   54 +++++++++++++++++++++++++++++++++-----------------
 transport.c         |    3 ++
 4 files changed, 53 insertions(+), 29 deletions(-)

diff --git a/builtin-fetch.c b/builtin-fetch.c
index 817dd6b..b51a019 100644
--- a/builtin-fetch.c
+++ b/builtin-fetch.c
@@ -346,12 +346,17 @@ static int store_updated_refs(const char *raw_url, const char *remote_name,
 			what = rm->name;
 		}
 
-		url_len = strlen(url);
-		for (i = url_len - 1; url[i] == '/' && 0 <= i; i--)
-			;
-		url_len = i + 1;
-		if (4 < i && !strncmp(".git", url + i - 3, 4))
-			url_len = i - 3;
+		if (url) {
+			url_len = strlen(url);
+			for (i = url_len - 1; url[i] == '/' && 0 <= i; i--)
+				;
+			url_len = i + 1;
+			if (4 < i && !strncmp(".git", url + i - 3, 4))
+				url_len = i - 3;
+		} else {
+			url = "foreign";
+			url_len = strlen(url);
+		}
 
 		note_len = 0;
 		if (*what) {
@@ -663,7 +668,7 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)
 	if (!remote)
 		die("Where do you want to fetch from today?");
 
-	transport = transport_get(remote, remote->url[0]);
+	transport = transport_get(remote, NULL);
 	if (verbosity >= 2)
 		transport->verbose = 1;
 	if (verbosity < 0)
diff --git a/builtin-ls-remote.c b/builtin-ls-remote.c
index 78a88f7..c6df8cf 100644
--- a/builtin-ls-remote.c
+++ b/builtin-ls-remote.c
@@ -87,9 +87,9 @@ int cmd_ls_remote(int argc, const char **argv, const char *prefix)
 		}
 	}
 	remote = nongit ? NULL : remote_get(dest);
-	if (remote && !remote->url_nr)
-		die("remote %s has no configured URL", dest);
-	transport = transport_get(remote, remote ? remote->url[0] : dest);
+	if (!nongit && !remote)
+		die("remote %s has invalid configuration", dest);
+	transport = transport_get(remote, remote ? NULL : dest);
 	if (uploadpack != NULL)
 		transport_set_option(transport, TRANS_OPT_UPLOADPACK, uploadpack);
 
diff --git a/builtin-push.c b/builtin-push.c
index 1d92e22..ae63521 100644
--- a/builtin-push.c
+++ b/builtin-push.c
@@ -88,6 +88,26 @@ static void setup_default_push_refspecs(void)
 	}
 }
 
+static int push_with_options(struct transport *transport, int flags)
+{
+	int err;
+	if (receivepack)
+		transport_set_option(transport,
+				     TRANS_OPT_RECEIVEPACK, receivepack);
+	if (thin)
+		transport_set_option(transport, TRANS_OPT_THIN, "yes");
+
+	if (flags & TRANSPORT_PUSH_VERBOSE)
+		fprintf(stderr, "Pushing to %s\n", transport->url);
+	err = transport_push(transport, refspec_nr, refspec, flags);
+	err |= transport_disconnect(transport);
+
+	if (!err)
+		return 0;
+
+	return 1;
+}
+
 static int do_push(const char *repo, int flags)
 {
 	int i, errs;
@@ -136,26 +156,22 @@ static int do_push(const char *repo, int flags)
 		url = remote->url;
 		url_nr = remote->url_nr;
 	}
-	for (i = 0; i < url_nr; i++) {
+	if (url_nr) {
+		for (i = 0; i < url_nr; i++) {
+			struct transport *transport =
+				transport_get(remote, url[i]);
+			if (push_with_options(transport, flags)) {
+				error("failed to push some refs to '%s'", url[i]);
+				errs++;
+			}
+		}
+	} else {
 		struct transport *transport =
-			transport_get(remote, url[i]);
-		int err;
-		if (receivepack)
-			transport_set_option(transport,
-					     TRANS_OPT_RECEIVEPACK, receivepack);
-		if (thin)
-			transport_set_option(transport, TRANS_OPT_THIN, "yes");
-
-		if (flags & TRANSPORT_PUSH_VERBOSE)
-			fprintf(stderr, "Pushing to %s\n", url[i]);
-		err = transport_push(transport, refspec_nr, refspec, flags);
-		err |= transport_disconnect(transport);
-
-		if (!err)
-			continue;
-
-		error("failed to push some refs to '%s'", url[i]);
-		errs++;
+			transport_get(remote, NULL);
+		if (push_with_options(transport, flags)) {
+			error("failed to push some refs to foreign system");
+			errs++;
+		}
 	}
 	return !!errs;
 }
diff --git a/transport.c b/transport.c
index f414bd3..7ba7ea9 100644
--- a/transport.c
+++ b/transport.c
@@ -926,6 +926,9 @@ struct transport *transport_get(struct remote *remote, const char *url)
 	struct transport *ret = xcalloc(1, sizeof(*ret));
 
 	ret->remote = remote;
+
+	if (!url && remote && remote->url)
+		url = remote->url[0];
 	ret->url = url;
 
 	if (remote && remote->foreign_vcs) {
-- 
1.6.4.rc3.138.ga6b98.dirty

[RFCv2 07/12] Add a transport implementation using git-vcs-* helpers

[Johan Herland]

From: Daniel Barkalow <barkalow@iabervon.org>

This is somewhat careless about pushes (that is, it attempts to make
pushes that the helper can't necessarily handle), but actually works for
fetches and simple pushes.

Signed-off-by: Daniel Barkalow <barkalow@iabervon.org>
Signed-off-by: Johan Herland <johan@herland.net>
---
 Makefile            |    1 +
 remote.c            |    2 +-
 transport-foreign.c |  200 +++++++++++++++++++++++++++++++++++++++++++++++++++
 transport.c         |    1 +
 transport.h         |    3 +
 5 files changed, 206 insertions(+), 1 deletions(-)
 create mode 100644 transport-foreign.c

diff --git a/Makefile b/Makefile
index c9003d7..fe62e2b 100644
--- a/Makefile
+++ b/Makefile
@@ -553,6 +553,7 @@ LIB_OBJS += symlinks.o
 LIB_OBJS += tag.o
 LIB_OBJS += trace.o
 LIB_OBJS += transport.o
+LIB_OBJS += transport-foreign.o
 LIB_OBJS += tree-diff.o
 LIB_OBJS += tree.o
 LIB_OBJS += tree-walk.o
diff --git a/remote.c b/remote.c
index 106151f..057ac02 100644
--- a/remote.c
+++ b/remote.c
@@ -50,7 +50,7 @@ static char buffer[BUF_SIZE];
 
 static int valid_remote(const struct remote *remote)
 {
-	return !!remote->url;
+	return remote->url || remote->foreign_vcs;
 }
 
 static const char *alias_url(const char *url)
diff --git a/transport-foreign.c b/transport-foreign.c
new file mode 100644
index 0000000..29aad77
--- /dev/null
+++ b/transport-foreign.c
@@ -0,0 +1,200 @@
+#include "cache.h"
+#include "transport.h"
+
+#include "run-command.h"
+#include "commit.h"
+#include "diff.h"
+#include "revision.h"
+
+struct foreign_data
+{
+	struct child_process *importer;
+};
+
+static struct child_process *get_importer(struct transport *transport)
+{
+	struct child_process *importer = transport->data;
+	if (!importer) {
+		struct strbuf buf;
+		importer = xcalloc(1, sizeof(*importer));
+		importer->in = -1;
+		importer->out = -1;
+		importer->err = 0;
+		importer->argv = xcalloc(3, sizeof(*importer->argv));
+		strbuf_init(&buf, 80);
+		strbuf_addf(&buf, "vcs-%s", transport->remote->foreign_vcs);
+		importer->argv[0] = buf.buf;
+		importer->argv[1] = transport->remote->name;
+		importer->git_cmd = 1;
+		start_command(importer);
+		transport->data = importer;
+	}
+	return importer;
+}
+
+static int disconnect_foreign(struct transport *transport)
+{
+	struct child_process *importer = transport->data;
+	if (importer) {
+		write(importer->in, "\n", 1);
+		close(importer->in);
+		finish_command(importer);
+		free(importer);
+		transport->data = NULL;
+	}
+	return 0;
+}
+
+static int fetch_refs_via_foreign(struct transport *transport,
+				  int nr_heads, struct ref **to_fetch)
+{
+	struct child_process *importer;
+	struct child_process fastimport;
+	struct ref *posn;
+	int i, count;
+
+	count = 0;
+	for (i = 0; i < nr_heads; i++) {
+		posn = to_fetch[i];
+		if (posn->status & REF_STATUS_UPTODATE)
+			continue;
+		count++;
+	}
+	if (count) {
+		importer = get_importer(transport);
+
+		memset(&fastimport, 0, sizeof(fastimport));
+		fastimport.in = importer->out;
+		fastimport.argv = xcalloc(3, sizeof(*fastimport.argv));
+		fastimport.argv[0] = "fast-import";
+		fastimport.argv[1] = "--quiet";
+		fastimport.git_cmd = 1;
+		start_command(&fastimport);
+
+		for (i = 0; i < nr_heads; i++) {
+			posn = to_fetch[i];
+			if (posn->status & REF_STATUS_UPTODATE)
+				continue;
+			write(importer->in, "import ", 7);
+			write(importer->in, posn->name, strlen(posn->name));
+			write(importer->in, "\n", 1);
+		}
+		disconnect_foreign(transport);
+		finish_command(&fastimport);
+	}
+	for (i = 0; i < nr_heads; i++) {
+		posn = to_fetch[i];
+		if (posn->status & REF_STATUS_UPTODATE)
+			continue;
+		read_ref(posn->name, posn->old_sha1);
+	}
+	return 0;
+}
+
+static struct ref *get_refs_via_foreign(struct transport *transport, int for_push)
+{
+	struct child_process *importer;
+	struct ref *ret = NULL;
+	struct ref **end = &ret;
+	struct strbuf buf;
+	FILE *file;
+
+	importer = get_importer(transport);
+	write(importer->in, "list\n", 5);
+
+	strbuf_init(&buf, 0);
+	file = fdopen(importer->out, "r");
+	while (1) {
+		char *eon;
+		if (strbuf_getline(&buf, file, '\n') == EOF)
+			break;
+
+		if (!*buf.buf)
+			break;
+
+		eon = strchr(buf.buf, ' ');
+		if (eon)
+			*eon = '\0';
+		*end = alloc_ref(buf.buf);
+		if (eon) {
+			if (strstr(eon + 1, "unchanged")) {
+				(*end)->status |= REF_STATUS_UPTODATE;
+				if (read_ref((*end)->name, (*end)->old_sha1))
+					die("Unchanged?");
+				fprintf(stderr, "Old: %p %s\n", *end, sha1_to_hex((*end)->old_sha1));
+			}
+		}
+		end = &((*end)->next);
+		strbuf_reset(&buf);
+	}
+
+	strbuf_release(&buf);
+	return ret;
+}
+
+static int foreign_push(struct transport *transport, struct ref *remote_refs, int flags) {
+	struct ref *ref, *has;
+	struct child_process *importer;
+	struct rev_info revs;
+	struct commit *commit;
+	struct child_process fastimport;
+
+	importer = get_importer(transport);
+
+	memset(&fastimport, 0, sizeof(fastimport));
+	fastimport.in = importer->out;
+	fastimport.argv = xcalloc(3, sizeof(*fastimport.argv));
+	fastimport.argv[0] = "fast-import";
+	fastimport.argv[1] = "--quiet";
+	fastimport.git_cmd = 1;
+	start_command(&fastimport);
+	for (ref = remote_refs; ref; ref = ref->next) {
+		if (!ref->peer_ref) {
+			ref->status = REF_STATUS_NONE;
+			continue;
+		}
+		init_revisions(&revs, NULL);
+		revs.reverse = 1;
+		for (has = remote_refs; has; has = has->next) {
+			commit = lookup_commit(has->old_sha1);
+			commit->object.flags |= UNINTERESTING;
+			add_pending_object(&revs, &commit->object, has->name);
+		}
+		commit = lookup_commit(ref->peer_ref->new_sha1);
+		add_pending_object(&revs, &commit->object, ref->name);
+
+		if (prepare_revision_walk(&revs))
+			die("Something wrong");
+
+		ref->status = REF_STATUS_UPTODATE;
+		while ((commit = get_revision(&revs))) {
+			ref->status = REF_STATUS_EXPECTING_REPORT;
+			fprintf(stderr, "export %s %s\n", sha1_to_hex(commit->object.sha1), ref->name);
+			write(importer->in, "export ", 7);
+			write(importer->in, sha1_to_hex(commit->object.sha1), 40);
+			write(importer->in, " ", 1);
+			write(importer->in, ref->name, strlen(ref->name));
+			write(importer->in, "\n", 1);
+		}
+	}
+
+	disconnect_foreign(transport);
+	finish_command(&fastimport);
+
+	for (ref = remote_refs; ref; ref = ref->next) {
+		read_ref(ref->name, ref->new_sha1);
+		if (ref->status == REF_STATUS_EXPECTING_REPORT)
+			ref->status = REF_STATUS_OK;
+	}
+
+	return 0;
+}
+
+void transport_foreign_init(struct transport *transport)
+{
+	transport->get_refs_list = get_refs_via_foreign;
+	transport->fetch = fetch_refs_via_foreign;
+	transport->push_refs = foreign_push;
+	transport->disconnect = disconnect_foreign;
+	transport->url = transport->remote->foreign_vcs;
+}
diff --git a/transport.c b/transport.c
index 7ba7ea9..643f20e 100644
--- a/transport.c
+++ b/transport.c
@@ -932,6 +932,7 @@ struct transport *transport_get(struct remote *remote, const char *url)
 	ret->url = url;
 
 	if (remote && remote->foreign_vcs) {
+		transport_foreign_init(ret);
 	} else if (!prefixcmp(url, "rsync:")) {
 		ret->get_refs_list = get_refs_via_rsync;
 		ret->fetch = fetch_objs_via_rsync;
diff --git a/transport.h b/transport.h
index b45e6c5..190490e 100644
--- a/transport.h
+++ b/transport.h
@@ -115,4 +115,7 @@ void transport_unlock_pack(struct transport *transport);
 int transport_disconnect(struct transport *transport);
 char *transport_anonymize_url(const char *url);
 
+/* Transport methods defined outside transport.c */
+void transport_foreign_init(struct transport *transport);
+
 #endif
-- 
1.6.4.rc3.138.ga6b98.dirty

[RFCv2 08/12] Preliminary clarifications to git-vcs documentation

[Johan Herland]

Signed-off-by: Johan Herland <johan@herland.net>
---
 Documentation/git-vcs.txt |    7 +++++--
 1 files changed, 5 insertions(+), 2 deletions(-)

diff --git a/Documentation/git-vcs.txt b/Documentation/git-vcs.txt
index 402c927..febe415 100644
--- a/Documentation/git-vcs.txt
+++ b/Documentation/git-vcs.txt
@@ -22,13 +22,15 @@ capabilities (limited both by the particular helper and by the
 capabilities of the system they connect to), and they report what
 capabilities they support.
 
-These programs can store refs in refs/<system>/*, and arbitrary
-information in info/<system>.
+These programs can store refs in refs/<system>/*, note refs in
+refs/notes/<system>/*, and arbitrary information in info/<system>/*.
 
 COMMANDS
 --------
 
 Commands are given by the caller on the helper's standard input, one per line.
+The output of each command must be produced on the helper's standard output.
+The helper's standard error stream can be used for status/progress messages.
 
 'capabilities'::
 	Outputs a single line with a list of feature names separated
@@ -45,6 +47,7 @@ Commands are given by the caller on the helper's standard input, one per line.
 	operations, such as importing the content, to see whether it
 	matches.) Other information, not yet supported, may be output
 	as well, separated by single spaces.
+	The output list shall be terminated with a blank line.
 
 'import' ref::
 	Imports the given ref by outputting it in git-fast-import
-- 
1.6.4.rc3.138.ga6b98.dirty

[RFCv2 09/12] Teach foreign transport code to perform the "capabilities" command

[Johan Herland]

The features reported by the vcs helper are stored in the foreign_data struct
which is expanded for this purpose.

In the process, we also change the capabilities command slightly to expect
one feature per line (instead of all features on a single line). This enables
us to add features in the future that take one or more arguments. To terminate
the list of features, a blank line is output at the end.

Signed-off-by: Johan Herland <johan@herland.net>
---
 Documentation/git-vcs.txt |    8 ++--
 transport-foreign.c       |   83 ++++++++++++++++++++++++++++++++++-----------
 2 files changed, 67 insertions(+), 24 deletions(-)

diff --git a/Documentation/git-vcs.txt b/Documentation/git-vcs.txt
index febe415..85656d4 100644
--- a/Documentation/git-vcs.txt
+++ b/Documentation/git-vcs.txt
@@ -33,10 +33,10 @@ The output of each command must be produced on the helper's standard output.
 The helper's standard error stream can be used for status/progress messages.
 
 'capabilities'::
-	Outputs a single line with a list of feature names separated
-	by spaces. Each of these indicates a supported feature of the
-	helper, and the caller will only attempt operations that are
-	supported.
+	Outputs a list of feature names, one per line. Each of these
+	indicates a supported feature of the helper, and the caller
+	will only attempt operations that are supported.
+	The output list shall be terminated with a blank line.
 
 'list'::
 	Outputs the names of refs, one per line. These may be
diff --git a/transport-foreign.c b/transport-foreign.c
index 29aad77..be0a587 100644
--- a/transport-foreign.c
+++ b/transport-foreign.c
@@ -8,38 +8,81 @@
 
 struct foreign_data
 {
-	struct child_process *importer;
+	struct child_process importer;
+
+	/* capabilities */
+	unsigned export:1;
+	unsigned export_branch:1;
+	unsigned export_merges:1;
 };
 
+static void get_foreign_capabilities(struct foreign_data *fdata)
+{
+	struct strbuf buf;
+	FILE *file;
+
+	write(fdata->importer.in, "capabilities\n", 13);
+
+	strbuf_init(&buf, 0);
+	file = fdopen(fdata->importer.out, "r");
+	while (1) {
+		char *eon;
+		if (strbuf_getline(&buf, file, '\n') == EOF)
+			break;
+
+		if (!*buf.buf)
+			break;
+
+		eon = strchr(buf.buf, ' ');
+		if (eon)
+			*eon = '\0';
+
+		/* parse features */
+		if (!strcmp(buf.buf, "export"))
+			fdata->export = 1;
+		else if (!strcmp(buf.buf, "export-branch"))
+			fdata->export_branch = 1;
+		else if (!strcmp(buf.buf, "export-merges"))
+			fdata->export_merges = 1;
+		else
+			die("Invalid feature '%s'", buf.buf);
+
+		strbuf_reset(&buf);
+	}
+
+	strbuf_release(&buf);
+}
+
 static struct child_process *get_importer(struct transport *transport)
 {
-	struct child_process *importer = transport->data;
-	if (!importer) {
+	struct foreign_data *fdata = (struct foreign_data *) transport->data;
+	if (!fdata) {
 		struct strbuf buf;
-		importer = xcalloc(1, sizeof(*importer));
-		importer->in = -1;
-		importer->out = -1;
-		importer->err = 0;
-		importer->argv = xcalloc(3, sizeof(*importer->argv));
+		fdata = xcalloc(1, sizeof(*fdata));
+		fdata->importer.in = -1;
+		fdata->importer.out = -1;
+		fdata->importer.err = 0;
+		fdata->importer.argv = xcalloc(3, sizeof(*fdata->importer.argv));
 		strbuf_init(&buf, 80);
 		strbuf_addf(&buf, "vcs-%s", transport->remote->foreign_vcs);
-		importer->argv[0] = buf.buf;
-		importer->argv[1] = transport->remote->name;
-		importer->git_cmd = 1;
-		start_command(importer);
-		transport->data = importer;
+		fdata->importer.argv[0] = buf.buf;
+		fdata->importer.argv[1] = transport->remote->name;
+		fdata->importer.git_cmd = 1;
+		start_command(&fdata->importer);
+		get_foreign_capabilities(fdata);
+		transport->data = fdata;
 	}
-	return importer;
+	return &fdata->importer;
 }
 
 static int disconnect_foreign(struct transport *transport)
 {
-	struct child_process *importer = transport->data;
-	if (importer) {
-		write(importer->in, "\n", 1);
-		close(importer->in);
-		finish_command(importer);
-		free(importer);
+	struct foreign_data *fdata = (struct foreign_data *) transport->data;
+	if (fdata) {
+		write(fdata->importer.in, "\n", 1);
+		close(fdata->importer.in);
+		finish_command(&fdata->importer);
+		free(fdata);
 		transport->data = NULL;
 	}
 	return 0;
-- 
1.6.4.rc3.138.ga6b98.dirty

[RFCv2 10/12] Introduce a 'marks <filename>' feature to the foreign transport code

[Johan Herland]

The 'marks' feature is reported by the vcs helper when it requires the
fast-import marks database to loaded/saved by the git-fast-import process
that is executed by the foreign transport machinery. The feature is
advertised along with exactly one argument: the location of the file
containing the marks database.

Signed-off-by: Johan Herland <johan@herland.net>
---
 Documentation/git-vcs.txt |    8 ++++++++
 transport-foreign.c       |   32 ++++++++++++++++++++++++++++++--
 2 files changed, 38 insertions(+), 2 deletions(-)

diff --git a/Documentation/git-vcs.txt b/Documentation/git-vcs.txt
index 85656d4..75815ba 100644
--- a/Documentation/git-vcs.txt
+++ b/Documentation/git-vcs.txt
@@ -65,6 +65,14 @@ there may be other restrictions on what may be exported.
 FEATURES
 --------
 
+'marks' filename::
+	Helper requires the marks from a git-fast-import run to be loaded
+	from, and saved to, the given filename. When this "feature" is
+	advertised, each git-fast-import run must load and save the
+	internal marks database (see the --import-marks and --export-marks
+	option to git-fast-import for more details) located at the given
+	filename.
+
 'export'::
 	Helper supports exporting commits, at least exporting
 	non-merge commits whose parents are not the parents of any
diff --git a/transport-foreign.c b/transport-foreign.c
index be0a587..ae9cb9a 100644
--- a/transport-foreign.c
+++ b/transport-foreign.c
@@ -11,6 +11,7 @@ struct foreign_data
 	struct child_process importer;
 
 	/* capabilities */
+	char *marks_filename;
 	unsigned export:1;
 	unsigned export_branch:1;
 	unsigned export_merges:1;
@@ -38,7 +39,12 @@ static void get_foreign_capabilities(struct foreign_data *fdata)
 			*eon = '\0';
 
 		/* parse features */
-		if (!strcmp(buf.buf, "export"))
+		if (!strcmp(buf.buf, "marks")) {
+			if (!eon)
+				die("Feature 'marks' requires an argument");
+			fdata->marks_filename = xstrdup(eon + 1);
+		}
+		else if (!strcmp(buf.buf, "export"))
 			fdata->export = 1;
 		else if (!strcmp(buf.buf, "export-branch"))
 			fdata->export_branch = 1;
@@ -82,6 +88,7 @@ static int disconnect_foreign(struct transport *transport)
 		write(fdata->importer.in, "\n", 1);
 		close(fdata->importer.in);
 		finish_command(&fdata->importer);
+		free(fdata->marks_filename);
 		free(fdata);
 		transport->data = NULL;
 	}
@@ -91,10 +98,13 @@ static int disconnect_foreign(struct transport *transport)
 static int fetch_refs_via_foreign(struct transport *transport,
 				  int nr_heads, struct ref **to_fetch)
 {
+	struct foreign_data *fdata;
 	struct child_process *importer;
 	struct child_process fastimport;
 	struct ref *posn;
+	struct strbuf export_marks, import_marks;
 	int i, count;
+	FILE *f;
 
 	count = 0;
 	for (i = 0; i < nr_heads; i++) {
@@ -105,12 +115,28 @@ static int fetch_refs_via_foreign(struct transport *transport,
 	}
 	if (count) {
 		importer = get_importer(transport);
+		fdata = (struct foreign_data *) transport->data;
+		strbuf_init(&export_marks, 0);
+		strbuf_init(&import_marks, 0);
 
 		memset(&fastimport, 0, sizeof(fastimport));
 		fastimport.in = importer->out;
-		fastimport.argv = xcalloc(3, sizeof(*fastimport.argv));
+		fastimport.argv = xcalloc(5, sizeof(*fastimport.argv));
 		fastimport.argv[0] = "fast-import";
 		fastimport.argv[1] = "--quiet";
+		if (fdata->marks_filename) {
+			strbuf_addf(&export_marks, "--export-marks=%s",
+				fdata->marks_filename);
+			fastimport.argv[2] = export_marks.buf;
+
+			f = fopen(fdata->marks_filename, "r");
+			if (f) {
+				strbuf_addf(&import_marks, "--import-marks=%s",
+					fdata->marks_filename);
+				fastimport.argv[3] = import_marks.buf;
+				fclose(f);
+			}
+		}
 		fastimport.git_cmd = 1;
 		start_command(&fastimport);
 
@@ -124,6 +150,8 @@ static int fetch_refs_via_foreign(struct transport *transport,
 		}
 		disconnect_foreign(transport);
 		finish_command(&fastimport);
+		strbuf_release(&export_marks);
+		strbuf_release(&import_marks);
 	}
 	for (i = 0; i < nr_heads; i++) {
 		posn = to_fetch[i];
-- 
1.6.4.rc3.138.ga6b98.dirty

[RFCv2 12/12] Add simple test cases of git-vcs-cvs functionality

[Johan Herland]

Signed-off-by: Johan Herland <johan@herland.net>
---
 t/t9800-foreign-vcs-cvs-basic.sh |  518 ++++++++++++++++++++++++++++++++++++++
 t/t9801-foreign-vcs-cvs-fetch.sh |  291 +++++++++++++++++++++
 t/test-lib.sh                    |    1 +
 3 files changed, 810 insertions(+), 0 deletions(-)
 create mode 100755 t/t9800-foreign-vcs-cvs-basic.sh
 create mode 100755 t/t9801-foreign-vcs-cvs-fetch.sh

diff --git a/t/t9800-foreign-vcs-cvs-basic.sh b/t/t9800-foreign-vcs-cvs-basic.sh
new file mode 100755
index 0000000..78f5a98
--- /dev/null
+++ b/t/t9800-foreign-vcs-cvs-basic.sh
@@ -0,0 +1,518 @@
+#!/bin/sh
+
+test_description='git vcs-cvs basic tests'
+. ./test-lib.sh
+
+if ! test_have_prereq PYTHON; then
+	say 'skipping CVS foreign-vcs helper tests, python not available'
+	test_done
+fi
+
+CVS_EXEC=cvs
+CVS_OPTS="-f -q"
+CVS="$CVS_EXEC $CVS_OPTS"
+
+CVSROOT=$(pwd)/cvsroot
+export CVSROOT
+unset CVS_SERVER
+
+CVSMODULE=cvsmodule
+GITREMOTE=cvsremote
+
+if ! type $CVS_EXEC >/dev/null 2>&1
+then
+	say 'skipping vcs-cvs tests, $CVS_EXEC not found'
+	test_done
+fi
+
+test_expect_success 'setup cvsroot' '$CVS init'
+
+test_expect_success '#1: setup a cvs module' '
+
+	mkdir "$CVSROOT/$CVSMODULE" &&
+	$CVS co -d module-cvs $CVSMODULE &&
+	(
+		cd module-cvs &&
+		cat <<EOF >o_fortuna &&
+O Fortuna
+velut luna
+statu variabilis,
+
+semper crescis
+aut decrescis;
+vita detestabilis
+
+nunc obdurat
+et tunc curat
+ludo mentis aciem,
+
+egestatem,
+potestatem
+dissolvit ut glaciem.
+EOF
+		$CVS add o_fortuna &&
+		cat <<EOF >message &&
+add "O Fortuna" lyrics
+
+These public domain lyrics make an excellent sample text.
+EOF
+		$CVS commit -f -F message o_fortuna
+	)
+'
+
+test_expect_success 'set up CVS repo as a foreign remote' '
+
+	git config "user.name" "Test User"
+	git config "user.email" "test@example.com"
+	git config "remote.$GITREMOTE.vcs" cvs
+	git config "remote.$GITREMOTE.cvsRoot" "$CVSROOT"
+	git config "remote.$GITREMOTE.cvsModule" "$CVSMODULE"
+	git config "remote.$GITREMOTE.fetch" \
+		"+refs/cvs/$GITREMOTE/*:refs/remotes/$GITREMOTE/*"
+
+'
+
+test_expect_success '#1: git-vcs-cvs "capabilities" command' '
+
+	echo "capabilities" | git vcs-cvs "$GITREMOTE" > actual &&
+	cat <<EOF >expect &&
+marks .git/info/cvs/$GITREMOTE/marks
+
+EOF
+	test_cmp expect actual
+
+'
+
+test_expect_success '#1: git-vcs-cvs "list" command' '
+
+	echo "list" | git vcs-cvs "$GITREMOTE" > actual &&
+	cat <<EOF >expect &&
+refs/cvs/$GITREMOTE/HEAD changed
+
+EOF
+	test_cmp expect actual
+
+'
+
+test_expect_success '#1: git-vcs-cvs "import" command' '
+
+	echo "import refs/cvs/$GITREMOTE/HEAD" | git vcs-cvs "$GITREMOTE" > actual &&
+	cat <<EOF >expect &&
+# Importing CVS revision o_fortuna:1.1
+blob
+mark :1
+data 180
+O Fortuna
+velut luna
+statu variabilis,
+
+semper crescis
+aut decrescis;
+vita detestabilis
+
+nunc obdurat
+et tunc curat
+ludo mentis aciem,
+
+egestatem,
+potestatem
+dissolvit ut glaciem.
+
+commit refs/cvs/$GITREMOTE/HEAD
+mark :2
+data 82
+add "O Fortuna" lyrics
+
+These public domain lyrics make an excellent sample text.
+
+M 644 :1 o_fortuna
+
+# Importing note for object 2
+blob
+mark :3
+data 14
+o_fortuna:1.1
+
+commit refs/notes/cvs/$GITREMOTE
+mark :4
+data 43
+Annotate commits imported by "git vcs-cvs"
+
+N :3 :2
+
+blob
+mark :5
+data 32
+1 o_fortuna:1.1
+2 o_fortuna:1.1
+
+blob
+mark :6
+data 16
+blob 1
+commit 2
+
+commit refs/cvs/$GITREMOTE/_metadata
+mark :7
+data 39
+Updated metadata used by "git vcs-cvs"
+
+M 644 :5 CVS/marks
+M 644 :6 o_fortuna/1.1
+
+EOF
+	grep -v "^committer " actual > actual.filtered &&
+	test_cmp expect actual.filtered
+
+'
+
+test_expect_success '#1: Passing git-vcs-cvs output to git-fast-import' '
+
+	git fast-import --quiet \
+		--export-marks=".git/info/cvs/$GITREMOTE/marks" \
+		< actual &&
+	git gc
+
+'
+
+test_expect_success '#1: Verifying correctness of import' '
+
+	echo "verify HEAD" | git vcs-cvs "$GITREMOTE"
+
+'
+
+test_expect_success '#2: update cvs module' '
+
+	(
+		cd module-cvs &&
+		cat <<EOF >o_fortuna &&
+O Fortune,
+like the moon
+you are changeable,
+
+ever waxing
+and waning;
+hateful life
+
+first oppresses
+and then soothes
+as fancy takes it;
+
+poverty
+and power
+it melts them like ice.
+EOF
+		cat <<EOF >message &&
+translate to English
+
+My Latin is terrible.
+EOF
+		$CVS commit -f -F message o_fortuna
+	)
+'
+
+test_expect_success '#2: git-vcs-cvs "capabilities" command' '
+
+	echo "capabilities" | git vcs-cvs "$GITREMOTE" > actual &&
+	cat <<EOF >expect &&
+marks .git/info/cvs/$GITREMOTE/marks
+
+EOF
+	test_cmp expect actual
+
+'
+
+test_expect_success '#2: git-vcs-cvs "list" command' '
+
+	echo "list" | git vcs-cvs "$GITREMOTE" > actual &&
+	cat <<EOF >expect &&
+refs/cvs/$GITREMOTE/HEAD changed
+
+EOF
+	test_cmp expect actual
+
+'
+
+test_expect_success '#2: git-vcs-cvs "import" command' '
+
+	echo "import refs/cvs/$GITREMOTE/HEAD" | git vcs-cvs "$GITREMOTE" > actual &&
+	cat <<EOF >expect &&
+# Importing CVS revision o_fortuna:1.2
+blob
+mark :8
+data 179
+O Fortune,
+like the moon
+you are changeable,
+
+ever waxing
+and waning;
+hateful life
+
+first oppresses
+and then soothes
+as fancy takes it;
+
+poverty
+and power
+it melts them like ice.
+
+commit refs/cvs/$GITREMOTE/HEAD
+mark :9
+data 44
+translate to English
+
+My Latin is terrible.
+
+from refs/cvs/$GITREMOTE/HEAD^0
+M 644 :8 o_fortuna
+
+# Importing note for object 9
+blob
+mark :10
+data 14
+o_fortuna:1.2
+
+commit refs/notes/cvs/$GITREMOTE
+mark :11
+data 43
+Annotate commits imported by "git vcs-cvs"
+
+from refs/notes/cvs/$GITREMOTE^0
+N :10 :9
+
+blob
+mark :12
+data 32
+8 o_fortuna:1.2
+9 o_fortuna:1.2
+
+blob
+mark :13
+data 94
+
+blob
+mark :14
+data 16
+blob 8
+commit 9
+
+commit refs/cvs/$GITREMOTE/_metadata
+mark :15
+data 39
+Updated metadata used by "git vcs-cvs"
+
+from refs/cvs/$GITREMOTE/_metadata^0
+M 644 :12 CVS/marks
+M 644 :13 o_fortuna/1.1
+M 644 :14 o_fortuna/1.2
+
+EOF
+	grep -v -e "^committer " -e "\b[0-9a-f]\{40\}\b" actual > actual.filtered &&
+	test_cmp expect actual.filtered
+
+'
+
+test_expect_success '#2: Passing git-vcs-cvs output to git-fast-import' '
+
+	git fast-import --quiet \
+		--import-marks=".git/info/cvs/$GITREMOTE/marks" \
+		--export-marks=".git/info/cvs/$GITREMOTE/marks" \
+		< actual &&
+	git gc
+
+'
+
+test_expect_success '#2: Verifying correctness of import' '
+
+	echo "verify HEAD" | git vcs-cvs "$GITREMOTE"
+
+'
+
+test_expect_success '#3: update cvs module' '
+
+	(
+		cd module-cvs &&
+		echo 1 >tick &&
+		$CVS add tick &&
+		$CVS commit -f -m 1 tick
+	)
+
+'
+
+test_expect_success '#3: git-vcs-cvs "capabilities" command' '
+
+	echo "capabilities" | git vcs-cvs "$GITREMOTE" > actual &&
+	cat <<EOF >expect &&
+marks .git/info/cvs/$GITREMOTE/marks
+
+EOF
+	test_cmp expect actual
+
+'
+
+test_expect_success '#3: git-vcs-cvs "list" command' '
+
+	echo "list" | git vcs-cvs "$GITREMOTE" > actual &&
+	cat <<EOF >expect &&
+refs/cvs/$GITREMOTE/HEAD changed
+
+EOF
+	test_cmp expect actual
+
+'
+
+test_expect_success '#3: git-vcs-cvs "import" command' '
+
+	echo "import refs/cvs/$GITREMOTE/HEAD" | git vcs-cvs "$GITREMOTE" > actual &&
+	cat <<EOF >expect &&
+# Importing CVS revision tick:1.1
+blob
+mark :16
+data 2
+1
+
+commit refs/cvs/$GITREMOTE/HEAD
+mark :17
+data 2
+1
+
+from refs/cvs/$GITREMOTE/HEAD^0
+M 644 :16 tick
+
+# Importing note for object 17
+blob
+mark :18
+data 23
+o_fortuna:1.2
+tick:1.1
+
+commit refs/notes/cvs/$GITREMOTE
+mark :19
+data 43
+Annotate commits imported by "git vcs-cvs"
+
+from refs/notes/cvs/$GITREMOTE^0
+N :18 :17
+
+blob
+mark :20
+data 41
+16 tick:1.1
+17 tick:1.1
+17 o_fortuna:1.2
+
+blob
+mark :21
+data 104
+commit 17
+
+blob
+mark :22
+data 18
+blob 16
+commit 17
+
+commit refs/cvs/$GITREMOTE/_metadata
+mark :23
+data 39
+Updated metadata used by "git vcs-cvs"
+
+from refs/cvs/$GITREMOTE/_metadata^0
+M 644 :20 CVS/marks
+M 644 :21 o_fortuna/1.2
+M 644 :22 tick/1.1
+
+EOF
+	grep -v -e "^committer " -e "\b[0-9a-f]\{40\}\b" actual > actual.filtered &&
+	test_cmp expect actual.filtered
+
+'
+
+test_expect_success '#3: Passing git-vcs-cvs output to git-fast-import' '
+
+	git fast-import --quiet \
+		--import-marks=".git/info/cvs/$GITREMOTE/marks" \
+		--export-marks=".git/info/cvs/$GITREMOTE/marks" \
+		< actual &&
+	git gc
+
+'
+
+test_expect_success '#3: Verifying correctness of import' '
+
+	echo "verify HEAD" | git vcs-cvs "$GITREMOTE"
+
+'
+
+test_expect_success '#4: git-vcs-cvs "capabilities" command' '
+
+	echo "capabilities" | git vcs-cvs "$GITREMOTE" > actual &&
+	cat <<EOF >expect &&
+marks .git/info/cvs/$GITREMOTE/marks
+
+EOF
+	test_cmp expect actual
+
+'
+
+test_expect_success '#4: git-vcs-cvs "list" command' '
+
+	echo "list" | git vcs-cvs "$GITREMOTE" > actual &&
+	cat <<EOF >expect &&
+refs/cvs/$GITREMOTE/HEAD unchanged
+
+EOF
+	test_cmp expect actual
+
+'
+
+test_expect_success '#4: git-vcs-cvs "import" command' '
+
+	echo "import refs/cvs/$GITREMOTE/HEAD" | git vcs-cvs "$GITREMOTE" > actual &&
+	cat <<EOF >expect &&
+blob
+mark :24
+data 0
+
+blob
+mark :25
+data 142
+
+blob
+mark :26
+data 94
+
+commit refs/cvs/$GITREMOTE/_metadata
+mark :27
+data 39
+Updated metadata used by "git vcs-cvs"
+
+from refs/cvs/$GITREMOTE/_metadata^0
+M 644 :24 CVS/marks
+M 644 :25 o_fortuna/1.2
+M 644 :26 tick/1.1
+
+EOF
+	grep -v -e "^committer " -e "\b[0-9a-f]\{40\}\b" actual > actual.filtered &&
+	test_cmp expect actual.filtered
+
+'
+
+test_expect_success '#4: Passing git-vcs-cvs output to git-fast-import' '
+
+	git fast-import --quiet \
+		--import-marks=".git/info/cvs/$GITREMOTE/marks" \
+		--export-marks=".git/info/cvs/$GITREMOTE/marks" \
+		< actual &&
+	git gc
+
+'
+
+test_expect_success '#4: Verifying correctness of import' '
+
+	echo "verify HEAD" | git vcs-cvs "$GITREMOTE"
+
+'
+
+test_done
diff --git a/t/t9801-foreign-vcs-cvs-fetch.sh b/t/t9801-foreign-vcs-cvs-fetch.sh
new file mode 100755
index 0000000..62a2325
--- /dev/null
+++ b/t/t9801-foreign-vcs-cvs-fetch.sh
@@ -0,0 +1,291 @@
+#!/bin/sh
+
+test_description='git vcs-cvs basic tests'
+. ./test-lib.sh
+
+if ! test_have_prereq PYTHON; then
+	say 'skipping CVS foreign-vcs helper tests, python not available'
+	test_done
+fi
+
+CVS_EXEC=cvs
+CVS_OPTS="-f -q"
+CVS="$CVS_EXEC $CVS_OPTS"
+
+CVSROOT=$(pwd)/cvsroot
+export CVSROOT
+unset CVS_SERVER
+
+CVSMODULE=cvsmodule
+GITREMOTE=cvsremote
+
+if ! type $CVS_EXEC >/dev/null 2>&1
+then
+	say 'skipping vcs-cvs tests, $CVS_EXEC not found'
+	test_done
+fi
+
+verify () {
+	git log --reverse --format="--- %T%n%s%n%n%b" "$GITREMOTE/$1" >actual &&
+	test_cmp "expect.$1" actual &&
+	echo "verify $1" | git vcs-cvs "$GITREMOTE"
+}
+
+test_expect_success 'setup CVS repo' '$CVS init'
+
+test_expect_success 'create CVS module with initial commit' '
+
+	mkdir "$CVSROOT/$CVSMODULE" &&
+	$CVS co -d module-cvs $CVSMODULE &&
+	(
+		cd module-cvs &&
+		cat <<EOF >o_fortuna &&
+O Fortuna
+velut luna
+statu variabilis,
+
+semper crescis
+aut decrescis;
+vita detestabilis
+
+nunc obdurat
+et tunc curat
+ludo mentis aciem,
+
+egestatem,
+potestatem
+dissolvit ut glaciem.
+EOF
+		$CVS add o_fortuna &&
+		cat <<EOF >message &&
+add "O Fortuna" lyrics
+
+These public domain lyrics make an excellent sample text.
+EOF
+		$CVS commit -f -F message o_fortuna
+	)
+'
+
+test_expect_success 'set up CVS repo/module as a foreign remote' '
+
+	git config "user.name" "Test User"
+	git config "user.email" "test@example.com"
+	git config "remote.$GITREMOTE.vcs" cvs
+	git config "remote.$GITREMOTE.cvsRoot" "$CVSROOT"
+	git config "remote.$GITREMOTE.cvsModule" "$CVSMODULE"
+	git config "remote.$GITREMOTE.fetch" \
+		"+refs/cvs/$GITREMOTE/*:refs/remotes/$GITREMOTE/*"
+
+'
+
+test_expect_success 'initial fetch from CVS remote' '
+
+	cat <<EOF >expect.HEAD &&
+--- 0e06d780dedab23e683c686fb041daa9a84c936c
+add "O Fortuna" lyrics
+
+These public domain lyrics make an excellent sample text.
+
+EOF
+	git fetch "$GITREMOTE" &&
+	verify HEAD
+
+'
+
+test_expect_success 'CVS commit' '
+
+	(
+		cd module-cvs &&
+		cat <<EOF >o_fortuna &&
+O Fortune,
+like the moon
+you are changeable,
+
+ever waxing
+and waning;
+hateful life
+
+first oppresses
+and then soothes
+as fancy takes it;
+
+poverty
+and power
+it melts them like ice.
+EOF
+		cat <<EOF >message &&
+translate to English
+
+My Latin is terrible.
+EOF
+		$CVS commit -f -F message o_fortuna
+	) &&
+	cat <<EOF >>expect.HEAD &&
+--- daa87269a5e00388135ad9542dc16ab6754466e5
+translate to English
+
+My Latin is terrible.
+
+EOF
+	git fetch "$GITREMOTE" &&
+	verify HEAD
+
+'
+
+test_expect_success 'CVS commit with new file' '
+
+	(
+		cd module-cvs &&
+		echo 1 >tick &&
+		$CVS add tick &&
+		$CVS commit -f -m 1 tick
+	) &&
+	cat <<EOF >>expect.HEAD &&
+--- 486935b4fccecea9b64cbed3a797ebbcbe2b7461
+1
+
+
+EOF
+	git fetch "$GITREMOTE" &&
+	verify HEAD
+
+'
+
+test_expect_success 'fetch without CVS changes' '
+
+	git fetch "$GITREMOTE" &&
+	verify HEAD
+
+'
+
+test_expect_success 'add 2 CVS commits' '
+
+	(
+		cd module-cvs &&
+		echo 2 >tick &&
+		$CVS commit -f -m 2 tick &&
+		echo 3 >tick &&
+		$CVS commit -f -m 3 tick
+	) &&
+	cat <<EOF >>expect.HEAD &&
+--- 83437ab3e57bf0a42915de5310e3419792b5a36f
+2
+
+
+--- 60fc50406a82dc6bd32dc6e5f7bd23e4c3cdf7ef
+3
+
+
+EOF
+	git fetch "$GITREMOTE" &&
+	verify HEAD
+
+'
+
+test_expect_success 'CVS commit with removed file' '
+
+	(
+		cd module-cvs &&
+		$CVS remove -f tick &&
+		$CVS commit -f -m "remove file" tick
+	) &&
+	cat <<EOF >>expect.HEAD &&
+--- daa87269a5e00388135ad9542dc16ab6754466e5
+remove file
+
+
+EOF
+	git fetch "$GITREMOTE" &&
+	verify HEAD
+
+'
+
+test_expect_success 'CVS commit with several new files' '
+
+	(
+		cd module-cvs &&
+		echo spam >spam &&
+		echo sausage >sausage &&
+		echo eggs >eggs &&
+		$CVS add spam sausage eggs &&
+		$CVS commit -f -m "spam, sausage, and eggs" spam sausage eggs
+	) &&
+	cat <<EOF >>expect.HEAD &&
+--- 3190dfce44a6d5e9916b4870dbf8f37d1ca4ddaf
+spam, sausage, and eggs
+
+
+EOF
+	git fetch "$GITREMOTE" &&
+	verify HEAD
+
+'
+
+test_expect_success 'new CVS branch' '
+
+	(
+		cd module-cvs &&
+		$CVS tag -b foo
+	) &&
+	cp expect.HEAD expect.foo &&
+	git fetch "$GITREMOTE" &&
+	verify HEAD &&
+	verify foo
+
+'
+
+test_expect_success 'CVS commit on branch' '
+
+	(
+		cd module-cvs &&
+		$CVS up -r foo &&
+		echo "spam spam spam" >spam &&
+		$CVS commit -f -m "commit on branch foo" spam
+	) &&
+	cat <<EOF >>expect.foo &&
+--- 1aba123e5c83898ce3a8b976cc6064d60246aef4
+commit on branch foo
+
+
+EOF
+	git fetch "$GITREMOTE" &&
+	verify HEAD &&
+	verify foo
+
+'
+
+test_expect_success 'create CVS tag' '
+
+	(
+		cd module-cvs &&
+		$CVS tag bar
+	) &&
+	cp expect.foo expect.bar &&
+	git fetch "$GITREMOTE" &&
+	verify HEAD &&
+	verify foo &&
+	verify bar
+
+'
+
+test_expect_success 'another CVS commit on branch' '
+
+	(
+		cd module-cvs &&
+		echo "spam spam spam spam spam spam" >> spam &&
+		$CVS commit -f -m "another commit on branch foo" spam
+	) &&
+	cat <<EOF >>expect.foo &&
+--- 15a2635e76e8e5a5a8746021643de317452f2340
+another commit on branch foo
+
+
+EOF
+	git fetch "$GITREMOTE" &&
+	verify HEAD &&
+	verify foo &&
+	verify bar
+
+'
+
+test_done
diff --git a/t/test-lib.sh b/t/test-lib.sh
index 5fdc5d9..8eb8b95 100644
--- a/t/test-lib.sh
+++ b/t/test-lib.sh
@@ -706,6 +706,7 @@ case $(uname -s) in
 esac
 
 test -z "$NO_PERL" && test_set_prereq PERL
+test -z "$NO_PYTHON" && test_set_prereq PYTHON
 
 # test whether the filesystem supports symbolic links
 ln -s x y 2>/dev/null && test -h y 2>/dev/null && test_set_prereq SYMLINKS
-- 
1.6.4.rc3.138.ga6b98.dirty