fast-import tweaks for remote helpers (Re: Status of the svn remote helper project (Dec 2010, #1))

[Jonathan Nieder <jrnieder@gmail.com>]

Tomas Carnecky wrote:

> I simplified the code and the requirements on fast-import are much
> lighter now. All I need is a way to tell fast-import to stop writing
> refs and after each commit write its sha1 to stdout.

That's good to hear.  What should be the syntax for asking fast-import
not to write to a ref?  Something like this?

	commit
	mark :1
	committer c o mitter <committer@example.com> now
	data <<END
	...

Writing the sha1 as each commit is written: how early does the
frontend need access to the sha1?  Would a facility to report marks
back to the frontend at the end of the stream take care of it?

Based on [1] I guess the main need is some way for fast-import to
tell the transport machinery what refs the transport machinery
should update (or at least ought to report as updated).  A hackish
way might be to make the remote helper send "progress" commands
with that information.

> It's possible to
> to modify fast-import.c with a small patch to make it behave like
> that. However, I haven't followed the svn remote helper that much
> lately so I don't know whether one of the other patches already
> modifies fast-import in this way.

No, the patches have mostly been adding commands that send information
back to the frontend.

 cat-blob (<dataref> | <mark>):
	Sends back an old blob along with its length (in
	cat-file --batch format).  svn-fe uses this to acquire
	the preimage when applying deltas.

 ls <quoted-path>:
	Sends back information about the current state of a path
	in the commit being prepared (as a single line in ls-tree
	format).  svn-fe uses this to move around files and to find
	a <dataref> to use with cat-blob when applying deltas.

 ls (<dataref> | <mark>) <path>:
	Sends back information about a path in a previous revision
	(tag, commit, or tree), in ls-tree format.

 M 040000 (<dataref> | <mark>) <path>:
	Like "M 100644 <dataref> <path>", replaces an entry in the
	active commit with content of the frontend's choice.  This
	gets used to copy in old directories.

> From the beginning my code was meant to be just an example how the
> interaction between git and the svn remote helper could look like.

It makes a nice demo, too. :)

> For example I save the svn rev <-> sha1 mapping in notes, which is
> appears to work well. I'll take a look if I'll be able to use the
> svn-fe in my script.

svn-fe needs a fast mapping svn rev -> sha1; it currently uses a marks
file for that.  (In the back of my mind, I have the idea of using a
file that allows O(1) access, perhaps of the form

	<commit name for rev 1> NL
	<commit name for rev 2> NL
	...

but as Ram has noted, keeping the whole table in memory is pretty
cheap already.)  A remote helper needs a fast mapping sha1 -> svn rev,
and imho notes are ideal for that[2].

The way I imagine it, the authoritative mapping is in notes and the
reverse mapping (e.g. in a marks file) is rebuilt when needed.

[1] remote-helper branch at git://github.com/wereHamster/git.git
[2] Why?  When a project switches from one svn server to another,
revision numbers tend to change, so revision numbers are not permanent
enough to belong in the commit message imho.  (If only git-notes had
existed when git svn was written...)