[PATCH 4/5] Draft of API for git-vcs-*, transport.c code to use it.

[PATCH 4/5] Draft of API for git-vcs-*, transport.c code to use it.

[Daniel Barkalow]

---
 Documentation/git-vcs.txt |   93 ++++++++++++++++++++++++++++++++++++++
 transport.c               |  109 +++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 202 insertions(+), 0 deletions(-)
 create mode 100644 Documentation/git-vcs.txt

diff --git a/Documentation/git-vcs.txt b/Documentation/git-vcs.txt
new file mode 100644
index 0000000..fa02b76
--- /dev/null
+++ b/Documentation/git-vcs.txt
@@ -0,0 +1,93 @@
+git-vcs-*(1)
+============
+
+NAME
+----
+git-vcs-* - Helper programs for interoperation with foreign systems
+
+SYNOPSIS
+--------
+'git vcs-<system>' <command> [options] [arguments]
+
+DESCRIPTION
+-----------
+
+These programs are normally not used directly by end users, but are
+invoked by various git programs that interact with remote repositories
+when the repository they would operate on is part of a foreign
+system.
+
+Each 'git vcs-<system>' is a helper for interoperating with a
+particular version control system. Different helpers have different
+capabilities (limited both be the particular helper and by the
+capabilities of the system they connect to), and they report what
+capabilities they support.
+
+In general, these programs interact with a portion of the refs
+namespace that isn't used by the rest of git. The configuration will
+then (generally) map these refs into the remotes namespace. This
+allows the importer to do whatever it wants with its refs without
+affecting the state visible to normal git programs.
+
+COMMANDS
+--------
+
+'capabilities'::
+	Prints the capabilities of the helper, one per line. These are:
+	 - import: the basic import command
+	 - marks: import should be done with a saved marks file
+	 - find-new-branches: detect new branches
+	 - export: the general export command
+	 - fork: create a new branch and export to it
+	 - anonymous-fork: make commits on a branch without an inherent name
+	 - merge: merge branches (of whatever type the system supports)
+
+	If the helper doesn't support "merge", the default for pull is
+	to rebase instead of merging.
+
+'list'::
+	Takes the remote name, and outputs the names of refs. These
+	may be followed, after a single space, by "changed" or
+	"unchanged", indicating whether the foreign repository has
+	changed from the state in the ref. If the helper doesn't know,
+	it doesn't have to provide a value. (In particular, it
+	shouldn't do expensive operations, such as importing the
+	content, to see whether it matches.)
+
+'import'::
+	Takes the remote name and a list of names of refs, and imports
+	whatever it describes, by outputting it in git-fast-import
+	format.
+
+'export'::
+	Sends the branch to the foreign system and reimports it in
+	fast-import format.
+
+	Reads a list of commits from stdin, where each commit has no
+	parents which were neither produced by an earlier import nor
+	appearing earlier in the list, where some commit has the old
+	value of the branch as a parent, and where all commits listed
+	are ancestors of the last one. Furthermore:
+
+	 - if the system doesn't support merges, each of these commits
+	   has only a single parent;
+
+	 - if the system doesn't support anonymous branches, the first
+	   commit has the old value of the branch as a parent (if the
+	   branch already had a value), and all parents are either the
+	   commit listed immediately before or produced by an earlier
+	   import;
+
+	 - if the system doesn't support many-way merges, each commit
+	   has at most two parents.
+
+	export produces output in fast-import format giving the
+	content after a round-trip through the foreign system. This
+	also contains extra headers to report the mapping of original
+	git commits to reimported git commits (to facilitate rewriting
+	local branches to use the history-as-reimported instead of the
+	git-only version).
+
+	export reports how much it managed to export by producing
+	commits in the fast-import stream that replace the listed
+	items that were successfully exported.
diff --git a/transport.c b/transport.c
index 8a37db5..fe78169 100644
--- a/transport.c
+++ b/transport.c
@@ -916,6 +916,113 @@ static int disconnect_git(struct transport *transport)
 	return 0;
 }
 
+static int fetch_refs_via_foreign(struct transport *transport,
+				  int nr_heads, struct ref **to_fetch)
+{
+	struct remote *remote = transport->remote;
+	struct child_process importer;
+	struct child_process fastimport;
+	struct ref *posn;
+	int i, count;
+	struct strbuf buf;
+
+	memset(&importer, 0, sizeof(importer));
+	importer.in = 0;
+	importer.no_stdin = 1;
+	importer.out = -1;
+	importer.err = 0;
+	importer.argv = xcalloc(5 + nr_heads, sizeof(*importer.argv));
+	strbuf_init(&buf, 80);
+	strbuf_addf(&buf, "vcs-%s", remote->foreign_vcs);
+	importer.argv[0] = buf.buf;
+	importer.argv[1] = "import";
+	importer.argv[2] = remote->name;
+	count = 0;
+	for (i = 0; i < nr_heads; i++) {
+		posn = to_fetch[i];
+		if (posn->status & REF_STATUS_UPTODATE)
+			continue;
+		importer.argv[3 + count] = posn->name;
+		count++;
+	}
+	importer.git_cmd = 1;
+	if (count) {
+		start_command(&importer);
+
+		memset(&fastimport, 0, sizeof(fastimport));
+		fastimport.in = importer.out;
+		fastimport.argv = xcalloc(3, sizeof(*fastimport.argv));
+		fastimport.argv[0] = "fast-import";
+		fastimport.argv[1] = "--quiet";
+		fastimport.git_cmd = 1;
+		start_command(&fastimport);
+
+		finish_command(&importer);
+		finish_command(&fastimport);
+	}
+	strbuf_release(&buf);
+	for (i = 0; i < nr_heads; i++) {
+		posn = to_fetch[i];
+		if (posn->status & REF_STATUS_UPTODATE)
+			continue;
+		read_ref(posn->name, posn->old_sha1);
+		count++;
+	}
+	return 0;
+}
+
+static struct ref *get_refs_via_foreign(struct transport *transport)
+{
+	struct remote *remote = transport->remote;
+	struct child_process importer;
+	struct ref *ret = NULL;
+	struct ref **end = &ret;
+	struct strbuf buf;
+	memset(&importer, 0, sizeof(importer));
+	importer.in = 0;
+	importer.no_stdin = 1;
+	importer.out = -1;
+	importer.err = 0;
+	importer.argv = xcalloc(5, sizeof(*importer.argv));
+	strbuf_init(&buf, 80);
+	strbuf_addf(&buf, "vcs-%s", remote->foreign_vcs);
+	importer.argv[0] = buf.buf;
+	importer.argv[1] = "list";
+	importer.argv[2] = remote->name;
+	importer.git_cmd = 1;
+	start_command(&importer);
+
+	strbuf_reset(&buf);
+	while (1) {
+		char *eol, *eon;
+		if (strbuf_read(&buf, importer.out, 80) <= 0)
+			break;
+		while (1) {
+			eol = strchr(buf.buf, '\n');
+			if (!eol)
+				break;
+			*eol = '\0';
+			eon = strchr(buf.buf, ' ');
+			if (eon)
+				*eon = '\0';
+			*end = alloc_ref(buf.buf);
+			if (eon) {
+				if (strstr(eon + 1, "unchanged")) {
+					(*end)->status |= REF_STATUS_UPTODATE;
+					if (read_ref((*end)->name, (*end)->old_sha1))
+						die("Unchanged?");
+				}
+			}
+			end = &((*end)->next);
+			strbuf_remove(&buf, 0, eol - buf.buf + 1);
+		}
+	}
+
+	finish_command(&importer);
+	strbuf_release(&buf);
+	return ret;
+}
+
 static int is_local(const char *url)
 {
 	const char *colon = strchr(url, ':');
@@ -940,6 +1047,8 @@ struct transport *transport_get(struct remote *remote, const char *url)
 	ret->url = url;
 
 	if (remote && remote->foreign_vcs) {
+		ret->get_refs_list = get_refs_via_foreign;
+		ret->fetch = fetch_refs_via_foreign;
 	} else if (!prefixcmp(url, "rsync:")) {
 		ret->get_refs_list = get_refs_via_rsync;
 		ret->fetch = fetch_objs_via_rsync;
-- 
1.6.2.1.476.g9bf04b

Re: [PATCH 4/5] Draft of API for git-vcs-*, transport.c code to use it.

[Junio C Hamano]

Daniel Barkalow <barkalow@iabervon.org> writes:

> +'capabilities'::
> +	Prints the capabilities of the helper, one per line. These are:
> +	 - import: the basic import command
> +	 - marks: import should be done with a saved marks file
> +	 - find-new-branches: detect new branches
> +	 - export: the general export command
> +	 - fork: create a new branch and export to it
> +	 - anonymous-fork: make commits on a branch without an inherent name
> +	 - merge: merge branches (of whatever type the system supports)
> +
> +	If the helper doesn't support "merge", the default for pull is
> +	to rebase instead of merging.
> +
> +'list'::
> +	Takes the remote name, and outputs the names of refs. These
> +	may be followed, after a single space, by "changed" or
> +	"unchanged", indicating whether the foreign repository has
> +	changed from the state in the ref. If the helper doesn't know,
> +	it doesn't have to provide a value. (In particular, it
> +	shouldn't do expensive operations, such as importing the
> +	content, to see whether it matches.)

Does this roughly corresponds to get_remote_refs(), with "unchanged"
return turned into the current value of the ref while "changed" returned
as 0{40} in old_sha1 value?

For a vcs backend that lacks find-new-branches capability, when does the
set of refnames returned by this operation change?  Can the end user
request an expensive operation to make the list up-to-date?  Does the end
user need to?

> +'import'::
> +	Takes the remote name and a list of names of refs, and imports
> +	whatever it describes, by outputting it in git-fast-import
> +	format.
> +
> +'export'::
> +	Sends the branch to the foreign system and reimports it in
> +	fast-import format.

The above two description is inconsistent; say "git-fast-import" for both.

This seems to follow the model of git-svn in that we treat our history as
throw-away, export the history and give the authority to the other system
by discarding and replacing our history with whatever the other end gives
back to us by re-importing.  Because git is more flexible than anything
else, we could afford to do so, but I wonder if it is the right model and
mentality.

One downside is that you end up rebasing the git side by operating this
way, and a topology where multiple developers use one git repository as a
synchronization point and use that git repository to interface with the
foreign system becomes impossible.  Instead, these multiple developers
need to treat the foreign system as the central repository, and interface
with it individually.

Re: [PATCH 4/5] Draft of API for git-vcs-*, transport.c code to use it.

[Daniel Barkalow]

On Wed, 25 Mar 2009, Junio C Hamano wrote:

> Daniel Barkalow <barkalow@iabervon.org> writes:
> 
> > +'capabilities'::
> > +	Prints the capabilities of the helper, one per line. These are:
> > +	 - import: the basic import command
> > +	 - marks: import should be done with a saved marks file
> > +	 - find-new-branches: detect new branches
> > +	 - export: the general export command
> > +	 - fork: create a new branch and export to it
> > +	 - anonymous-fork: make commits on a branch without an inherent name
> > +	 - merge: merge branches (of whatever type the system supports)
> > +
> > +	If the helper doesn't support "merge", the default for pull is
> > +	to rebase instead of merging.
> > +
> > +'list'::
> > +	Takes the remote name, and outputs the names of refs. These
> > +	may be followed, after a single space, by "changed" or
> > +	"unchanged", indicating whether the foreign repository has
> > +	changed from the state in the ref. If the helper doesn't know,
> > +	it doesn't have to provide a value. (In particular, it
> > +	shouldn't do expensive operations, such as importing the
> > +	content, to see whether it matches.)
> 
> Does this roughly corresponds to get_remote_refs(), with "unchanged"
> return turned into the current value of the ref while "changed" returned
> as 0{40} in old_sha1 value?

Yes, with the [1/5] change to make 0{40} tell git to consider it different 
but unknown. Also, not giving either of these flags counts as 0{40}, but 
we might someday want to have the native side report the difference 
between knowing that it changed and not knowing at all.

> For a vcs backend that lacks find-new-branches capability, when does the
> set of refnames returned by this operation change?  Can the end user
> request an expensive operation to make the list up-to-date?  Does the end
> user need to?

What I've done for my p4 backend is have an option for whether it should 
search for new branches (which is somewhat expensive, at least currently, 
because it involves connecting to the server in this step). If the backend 
actually lacks the capability, the user has to do something (probably add 
them to the config file) to include new branches. In some cases, what the 
backend exposes as branches may be something that there isn't a complete 
list of; there may be systems that are structured sort of like repo.or.cz, 
where the URL is for the whole system, and there are some things on it 
that are branches of the project (like various people's git.git 
repositories) and some things that are completely unrelated and not 
interesting, and there's no machine-readable way to distinguish them.

> > +'import'::
> > +	Takes the remote name and a list of names of refs, and imports
> > +	whatever it describes, by outputting it in git-fast-import
> > +	format.
> > +
> > +'export'::
> > +	Sends the branch to the foreign system and reimports it in
> > +	fast-import format.
> 
> The above two description is inconsistent; say "git-fast-import" for both.

Not "fast-import" for both, now that other systems have adopted the format 
as input?

> This seems to follow the model of git-svn in that we treat our history as
> throw-away, export the history and give the authority to the other system
> by discarding and replacing our history with whatever the other end gives
> back to us by re-importing.  Because git is more flexible than anything
> else, we could afford to do so, but I wonder if it is the right model and
> mentality.

I think that there is always the requirement that the other system 
(whether it's foreign, git through filter-branch, or even a regular remote 
git repository) contains a sane, consistant, and normal-looking history. 
It's rarely going to be possible to export a history that will import as 
the same hash (and incremental imports on top of local history that isn't 
what a fresh complete import would contain seems even crazier). I don't 
think there's any particular use for using a foreign system to store 
history that only makes sense in git.

But, as a general principle, the reason a git developer would push to a 
non-git remote repository is because the remote repository is 
authoritative. I don't think it makes sense to have an environment where 
the authoritative history is in git, but people are sharing it through a 
bzr server (even if the bzr server can accurately represent the git 
history).

Of course, we may want a different mechanism some day to generalize 
git-cvsserver, where the authoritative history is in git, but we're 
supporting foreign clients. But that requires very different code from 
what I'm working on.

> One downside is that you end up rebasing the git side by operating this
> way, and a topology where multiple developers use one git repository as a
> synchronization point and use that git repository to interface with the
> foreign system becomes impossible.  Instead, these multiple developers
> need to treat the foreign system as the central repository, and interface
> with it individually.

Yeah, this is unfortunate. You can at least import through a shared 
system, which is pretty close to how git.git development goes (we pull 
from your public repo, make changes locally, and email patches, which you 
put in upstream, and we get back commits that are not the same hash we 
created locally).

I think it might be possible to do something where the synchronization 
point has a list of authoritative commit-swaps, where it can tell the 
multiple developers: "commit Y is really the same thing as commit X; they 
have the same tree, and their parents Y1..Yn are the same as X1..Xn 
(either based on the same criterion applied recursively, or having 
identical hashes)". This list of commit-swaps allows the developers to be 
comfortable with trivial rebases (in a commit Z with X as a parent, make a 
commit Z' with Y replacing X and no other changes, and replace Z with Z' 
in refs and other commits). Of course, this depends on being able to get 
the foreign system to agree on both complete content and history topology, 
but otherwise we kind of have to throw away our history, because the 
public history simply can't be like what we've made locally.

	-Daniel
*This .sig left intentionally blank*

Re: [PATCH 4/5] Draft of API for git-vcs-*, transport.c code to use it.

[Junio C Hamano]

Daniel Barkalow <barkalow@iabervon.org> writes:

>> > +'import'::
>> > +	Takes the remote name and a list of names of refs, and imports
>> > +	whatever it describes, by outputting it in git-fast-import
>> > +	format.
>> > +
>> > +'export'::
>> > +	Sends the branch to the foreign system and reimports it in
>> > +	fast-import format.
>> 
>> The above two description is inconsistent; say "git-fast-import" for both.
>
> Not "fast-import" for both, now that other systems have adopted the format 
> as input?

I did not mean anything deeper than just pointing out that one sentence
calls the format "git-fast-import format" and the other one calls
"fast-import format".

Re: [PATCH 4/5] Draft of API for git-vcs-*, transport.c code to use it.

[Junio C Hamano]

Daniel Barkalow <barkalow@iabervon.org> writes:

> On Wed, 25 Mar 2009, Junio C Hamano wrote:
>
>> This seems to follow the model of git-svn in that we treat our history as
>> throw-away, export the history and give the authority to the other system
>> by discarding and replacing our history with whatever the other end gives
>> back to us by re-importing.  Because git is more flexible than anything
>> else, we could afford to do so, but I wonder if it is the right model and
>> mentality.
> ...
> But, as a general principle, the reason a git developer would push to a
> non-git remote repository is because the remote repository is
> authoritative.  I don't think it makes sense to have an environment
> where the authoritative history is in git, but people are sharing it
> through a bzr server (even if the bzr server can accurately represent
> the git history).

Suppose a project used to use subversion, but it migrated to git (not an
unheard-of scenario these days, I hope).  The git repository now is the
authoritative one, all the development happens on this side.

But in order to help:

 - people who have established their workflow to follow the project
   (e.g. not necessarily contributing anything back, but just doing
   regular "svn update");

 - people who have leftover local changes from the subversion days; and

 - other project infrastracture (e.g. trac) that the project hasn't
   managed to interface to the new git repository yet;

the project decides to keep feeding recent updates to the subversion
repository that used to be authoritative, even though it is now declared
read-only (i.e. the only update comes from the git end).

My understanding is that the above scenario would not work if git-vcs-svn
rewrites commits when git exports to svn, and existing git-svn two-way
interface using its "dcommit" may have exactly the same issue.

The reason I brought this up was to involve people who have already faced
this issue with git-svn in the discussion to see if we can avoid it by
doing somethink similar to clever tricks they are using in their git-svn
workflow, if there are some.  Perhaps your paragraph below may be one of
those clever tricks, but there may be others.

> I think it might be possible to do something where the synchronization 
> point has a list of authoritative commit-swaps, where it can tell the 
> multiple developers: "commit Y is really the same thing as commit X; they 
> have the same tree, and their parents Y1..Yn are the same as X1..Xn 
> (either based on the same criterion applied recursively, or having 
> identical hashes)". This list of commit-swaps allows the developers to be 
> comfortable with trivial rebases (in a commit Z with X as a parent, make a 
> commit Z' with Y replacing X and no other changes, and replace Z with Z' 
> in refs and other commits). Of course, this depends on being able to get 
> the foreign system to agree on both complete content and history topology, 
> but otherwise we kind of have to throw away our history, because the 
> public history simply can't be like what we've made locally.

Re: [PATCH 4/5] Draft of API for git-vcs-*, transport.c code to use it.

[Daniel Barkalow]

On Wed, 25 Mar 2009, Junio C Hamano wrote:

> Suppose a project used to use subversion, but it migrated to git (not an
> unheard-of scenario these days, I hope).  The git repository now is the
> authoritative one, all the development happens on this side.
> 
> But in order to help:
> 
>  - people who have established their workflow to follow the project
>    (e.g. not necessarily contributing anything back, but just doing
>    regular "svn update");
> 
>  - people who have leftover local changes from the subversion days; and
> 
>  - other project infrastracture (e.g. trac) that the project hasn't
>    managed to interface to the new git repository yet;
> 
> the project decides to keep feeding recent updates to the subversion
> repository that used to be authoritative, even though it is now declared
> read-only (i.e. the only update comes from the git end).

Actually, this is easy: just configure the git repo to not fetch anything 
from the no-longer-authoritative subversion repository. git-vcs-svn would 
waste a bunch of time reimporting what it exported, but it wouldn't 
actually do anything with it (since it doesn't even have tracking refs to 
update). It could, of course, be optimized to avoid reimporting if it 
doesn't need to.

> My understanding is that the above scenario would not work if git-vcs-svn
> rewrites commits when git exports to svn, and existing git-svn two-way
> interface using its "dcommit" may have exactly the same issue.
> 
> The reason I brought this up was to involve people who have already faced
> this issue with git-svn in the discussion to see if we can avoid it by
> doing somethink similar to clever tricks they are using in their git-svn
> workflow, if there are some.  Perhaps your paragraph below may be one of
> those clever tricks, but there may be others.

The harder problem is that you can only push history that the remote 
system will be able to represent. In the same way that git wants your push 
to be a fast-forward, svn wants your push to not contain any merges (and 
p4 and svn both want your push to not contain parallel development 
pushed to a single branch). Rewriting the history to conform to the 
destination's requirements is most of what dcommit does, I believe.

I think that, in this use case, the right thing is to have a config option 
for git-vcs-svn to tell it to collapse the problematic commits (or maybe 
collapse all commits in a push). That would make the svn history export 
like the reflog, which is all linear and simple anyway, and probably 
sufficient for the above users of the old repository.

	-Daniel
*This .sig left intentionally blank*

Re: [PATCH 4/5] Draft of API for git-vcs-*, transport.c code to use it.

[Junio C Hamano]

Daniel Barkalow <barkalow@iabervon.org> writes:

> On Wed, 25 Mar 2009, Junio C Hamano wrote:
>
>> Suppose a project used to use subversion, but it migrated to git (not an
>> unheard-of scenario these days, I hope).  The git repository now is the
>> authoritative one, all the development happens on this side.
>> 
>> But in order to help:
>> 
>>  - people who have established their workflow to follow the project
>>    (e.g. not necessarily contributing anything back, but just doing
>>    regular "svn update");
>> 
>>  - people who have leftover local changes from the subversion days; and
>> 
>>  - other project infrastracture (e.g. trac) that the project hasn't
>>    managed to interface to the new git repository yet;
>> 
>> the project decides to keep feeding recent updates to the subversion
>> repository that used to be authoritative, even though it is now declared
>> read-only (i.e. the only update comes from the git end).
>
> Actually, this is easy: just configure the git repo to not fetch anything 
> from the no-longer-authoritative subversion repository. git-vcs-svn would 
> waste a bunch of time reimporting what it exported, but it wouldn't 
> actually do anything with it (since it doesn't even have tracking refs to 
> update). It could, of course, be optimized to avoid reimporting if it 
> doesn't need to.

I am afraid that won't fly; my comment that started this subthread was not
about your "import" but was about your "export" part.  It is about sending
the git branch to the other end, which is allowed to rewrite what we send
and force us to modify our history.

Re: [PATCH 4/5] Draft of API for git-vcs-*, transport.c code to use it.

[Daniel Barkalow]

On Wed, 25 Mar 2009, Junio C Hamano wrote:

> Daniel Barkalow <barkalow@iabervon.org> writes:
> 
> > On Wed, 25 Mar 2009, Junio C Hamano wrote:
> >
> >> Suppose a project used to use subversion, but it migrated to git (not an
> >> unheard-of scenario these days, I hope).  The git repository now is the
> >> authoritative one, all the development happens on this side.
> >> 
> >> But in order to help:
> >> 
> >>  - people who have established their workflow to follow the project
> >>    (e.g. not necessarily contributing anything back, but just doing
> >>    regular "svn update");
> >> 
> >>  - people who have leftover local changes from the subversion days; and
> >> 
> >>  - other project infrastracture (e.g. trac) that the project hasn't
> >>    managed to interface to the new git repository yet;
> >> 
> >> the project decides to keep feeding recent updates to the subversion
> >> repository that used to be authoritative, even though it is now declared
> >> read-only (i.e. the only update comes from the git end).
> >
> > Actually, this is easy: just configure the git repo to not fetch anything 
> > from the no-longer-authoritative subversion repository. git-vcs-svn would 
> > waste a bunch of time reimporting what it exported, but it wouldn't 
> > actually do anything with it (since it doesn't even have tracking refs to 
> > update). It could, of course, be optimized to avoid reimporting if it 
> > doesn't need to.
> 
> I am afraid that won't fly; my comment that started this subthread was not
> about your "import" but was about your "export" part.  It is about sending
> the git branch to the other end, which is allowed to rewrite what we send
> and force us to modify our history.

It can rewrite what we send, but if we're not reimporting it, we don't 
care what it's done. All we care about is that when we ignore the 
rewriting, git-vcs-* can still export more changes from our version of the 
history.

That is, if we push X, which gets rewritten to X' (but we ignore that), 
and we then push Y, whose parent is X (not X'), a high-quality git-vcs 
backend will be reliably able to accept the second push, just as if we'd 
created a Y' based on X'. And by "high-quality", I mean some testable 
capability with a better name.

	-Daniel
*This .sig left intentionally blank*