Announcing a new (prototype) git-remote-hg tool

Announcing a new (prototype) git-remote-hg tool

[Mike Hommey]

Hi,

I've (re)started work on a longstanding idea of mine of having a git tool
talking the mercurial wire protocol directly. I'm now at a stage where
the tool can clone and pull/fetch from mercurial.

As it is a prototype, there are many things that it doesn't handle (like
named branches, bookmarks, phases, obsolescence markers), but it
currently transposes a complete mercurial dag to git and maintains
metadata about the original mercurial data.

Code on https://github.com/glandium/git-remote-hg

It doesn't support push, but support for that should come in the coming
weeks.

More background on http://glandium.org/blog/?p=3382

This is meant to be a prototype, and will stay that way for now.
It's a validation that this can actually work. Now that I have pull
support I know I can make it push.
I'm currently evaluating what the final tool would look like. I'm *very*
tempted to implement it in C, based on core git code, because there are
many things that this helper does that would be so much easier to do
with direct access to git's guts. And that wouldn't require more
dependencies than git currently has: it would "just" need curl and ssh,
and git already uses both.

If I were to go in that direction, would you consider integrating it
in git core? If not, would you rather see git helpers to make this
git-remote-hg helper more efficient?

Cheers,

Mike

Re: Announcing a new (prototype) git-remote-hg tool

[Jonathan Nieder]

Mike Hommey wrote:

> I'm currently evaluating what the final tool would look like. I'm *very*
> tempted to implement it in C, based on core git code, because there are
> many things that this helper does that would be so much easier to do
> with direct access to git's guts. And that wouldn't require more
> dependencies than git currently has: it would "just" need curl and ssh,
> and git already uses both.
>
> If I were to go in that direction, would you consider integrating it
> in git core?

Yes --- I would like this a lot.

The general trend has been to carry fewer contrib-style tools in-tree,
since the problem of discovering tools built on top of git is not as
hard as it used to be.  What you describe above seems to be a bit of an
exception:

 - libgit.a in its current state evolves too quickly for it to be
   convenient for out-of-tree tools to use.  cgit <http://git.zx2c4.com/cgit/>
   uses git pinned to a particular version as a submodule to get around
   this, which is fussy and has bad implications for remembering to
   get security updates.

 - an in-tree user of libgit.a would be useful as a reference example
   to use to try to make libgit.a into be a better library internally
   (and eventually expose e.g. by merging with libgit2 as something
   outside tools can link to, I hope)

 - if it makes sense to help people using the current remote helper
   in contrib to migrate to this, it could be convenient for users

In other words, although in the long term I would be happiest if
libgit becomes good enough to let this project live in a separate tree
and link to it, it's tempting to build this in-tree because we're not
there yet.

Some other alternatives:

 - using libgit2 <https://libgit2.github.com/>

 - improving git plumbing (e.g., with new fast-import commands) or
   exposing a small library with a stable API for the tool's use

I haven't thought it through carefully but at the moment I like the
in-tree approach best.

Thanks,
Jonathan

Re: Announcing a new (prototype) git-remote-hg tool

[Jeff King]

On Fri, Dec 05, 2014 at 02:13:19PM -0800, Jonathan Nieder wrote:

> Mike Hommey wrote:
> 
> > I'm currently evaluating what the final tool would look like. I'm *very*
> > tempted to implement it in C, based on core git code, because there are
> > many things that this helper does that would be so much easier to do
> > with direct access to git's guts. And that wouldn't require more
> > dependencies than git currently has: it would "just" need curl and ssh,
> > and git already uses both.
> >
> > If I were to go in that direction, would you consider integrating it
> > in git core?
> 
> Yes --- I would like this a lot.

I'm concerned that this tool will have drawbacks that Felipe's remote-hg
does not. And I can well imagine that it may, as that tool builds on
Mercurial's API, which will probably handle some corner cases
differently. This isn't to disparage Mike's attempt; it will probably
have some upsides, too. But given that the approaches are so different,
it does not seem obvious to me that one will always be better than the
other.

One of the nice things about spinning remote-hg out of the core repo is
that it means we do not have to endorse a particular implementation, and
they can compete with each other on their merits.  I would very much
hate to see Felipe's remote-hg project wither and die just because
another implementation becomes the de facto standard by being included
in git.git. It's a proven tool, and this new thing is not yet.

It's a shame that both squat on the name "remote-hg", because it makes
it difficult to tell the two apart. But of course that is the only way
to make "git clone hg::..." work. Maybe we need a layer of indirection?
:)

>  - libgit.a in its current state evolves too quickly for it to be
>    convenient for out-of-tree tools to use.  cgit <http://git.zx2c4.com/cgit/>
>    uses git pinned to a particular version as a submodule to get around
>    this, which is fussy and has bad implications for remembering to
>    get security updates.

I'm not sure that this approach is any better than carrying something in
contrib/ in git.git. If I refactor a function in libgit.a, I notice
breakage in the callers because it no longer compiles, or because I am
thorough and look at the implications to git callers.

I do _not_ want to be responsible for making sure that contrib/* still
builds. That is the problem of the maintainer of the contrib/ project in
question. That may sound a little selfish, but I think that is what it
means to be in contrib, and not in the regular tree.

So once you realize that is the burden of the contrib/ author to fix
breakages, then the process is:

  git pull
  cd contrib/c-remote-hg
  make
  # oops, it broke
  fix fix fix

That is not any different than:

  git submodule add git...
  git submodule update
  make
  # oops, it broke
  fix fix fix

The hard part is not how you pull changes from the new git into your
tree. It is the fact that upstream may be breaking the interface behind
your back.  And your best bet is to aggressively merge with upstream,
rather than trying to track only occasional release versions.

Of course, if you meant to _really_ carry it in-tree, not in contrib/,
then none of that applies. But then I worry doubly about the
"endorsement" issue.

-Peff

Re: Announcing a new (prototype) git-remote-hg tool

[Jonathan Nieder]

Jeff King wrote:

> One of the nice things about spinning remote-hg out of the core repo is
> that it means we do not have to endorse a particular implementation, and
> they can compete with each other on their merits.

True.

[...]
> It's a shame that both squat on the name "remote-hg", because it makes
> it difficult to tell the two apart. But of course that is the only way
> to make "git clone hg::..." work. Maybe we need a layer of indirection?
> :)

If the helpers are roughly interchangeable (that is, if you can switch
between fetching using each one into the same on-disk git repository),
then picking one to symlink as git-remote-hg in your $PATH should be
enough.

If they don't have that level of interoperability, then there's an
argument to be made that the URLs shouldn't be the same.
Unfortunately url.*.insteadof rules are resolved at fetch time instead
of being resolved once and the result recorded in .git/config.  So
yes, it seems like a way to have abbreviations for URLs (e.g., hg::
meaning hg+mh:: or hg+fc::) that get resolved at clone time would be
useful.  It's a layer of indirection we don't provide. :/

Thanks,
Jonathan

Re: Announcing a new (prototype) git-remote-hg tool

[Mike Hommey]

On Fri, Dec 05, 2014 at 03:13:30PM -0800, Jonathan Nieder wrote:
> Jeff King wrote:
> 
> > One of the nice things about spinning remote-hg out of the core repo is
> > that it means we do not have to endorse a particular implementation, and
> > they can compete with each other on their merits.
> 
> True.
> 
> [...]
> > It's a shame that both squat on the name "remote-hg", because it makes
> > it difficult to tell the two apart. But of course that is the only way
> > to make "git clone hg::..." work. Maybe we need a layer of indirection?
> > :)
> 
> If the helpers are roughly interchangeable (that is, if you can switch
> between fetching using each one into the same on-disk git repository),
> then picking one to symlink as git-remote-hg in your $PATH should be
> enough.
> 
> If they don't have that level of interoperability, then there's an
> argument to be made that the URLs shouldn't be the same.

I don't think Felipe's and the one that uses hg-git under the hood are
already interoperable. Mine is also different from both. They should all
produce the same git trees. They don't produce the same commits. They
also don't share the same metadata.

Mike

Re: Announcing a new (prototype) git-remote-hg tool

[Jeff King]

On Fri, Dec 05, 2014 at 03:13:30PM -0800, Jonathan Nieder wrote:

> > It's a shame that both squat on the name "remote-hg", because it makes
> > it difficult to tell the two apart. But of course that is the only way
> > to make "git clone hg::..." work. Maybe we need a layer of indirection?
> > :)
> 
> If the helpers are roughly interchangeable (that is, if you can switch
> between fetching using each one into the same on-disk git repository),
> then picking one to symlink as git-remote-hg in your $PATH should be
> enough.

That may be enough. For the most part you do not need to agree with
other members of the project on which implementation to use. My
experience with import tools has been that either:

  1. you are using them personally (because you do not like the
     upstream's choice of VCS and would prefer to transparently work in
     your favorite tool), or

  2. there is a group of developers who want to use git, but
     somebody provides an unofficial git mirror. They do not have to
     agree on the tool, because they just use git directly from the
     mirror.

So it is mostly a personal choice. But the two confusions I'd still
anticipate are:

  a. It's difficult to even _talk_ about the tools, because the names
     are the same (so searching for tips on the tool, reporting bugs,
     etc, are harder than necessary).

  b. You may want different tools for different projects. If one tool is
     much more efficient, you may need it for a large repo (e.g.,
     mozilla). But another tool may provide other features, and you
     would prefer it for smaller repos.

This is largely speculation, though, and I do not actively use the tools
myself. So I'd be happy to push off dealing with it until it itches
enough for somebody to scratch.

-Peff

Re: Announcing a new (prototype) git-remote-hg tool

[Mike Hommey]

On Fri, Dec 05, 2014 at 05:59:30PM -0500, Jeff King wrote:
> On Fri, Dec 05, 2014 at 02:13:19PM -0800, Jonathan Nieder wrote:
> 
> > Mike Hommey wrote:
> > 
> > > I'm currently evaluating what the final tool would look like. I'm *very*
> > > tempted to implement it in C, based on core git code, because there are
> > > many things that this helper does that would be so much easier to do
> > > with direct access to git's guts. And that wouldn't require more
> > > dependencies than git currently has: it would "just" need curl and ssh,
> > > and git already uses both.
> > >
> > > If I were to go in that direction, would you consider integrating it
> > > in git core?
> > 
> > Yes --- I would like this a lot.
> 
> I'm concerned that this tool will have drawbacks that Felipe's remote-hg
> does not. And I can well imagine that it may, as that tool builds on
> Mercurial's API, which will probably handle some corner cases
> differently.

FWIW, my tool only uses the mercurial code for the wire protocol. This
can (and if I go the C route, will) be implemented without using
mercurial code, it's really not a hard problem.

> This isn't to disparage Mike's attempt; it will probably
> have some upsides, too. But given that the approaches are so different,
> it does not seem obvious to me that one will always be better than the
> other.
> 
> One of the nice things about spinning remote-hg out of the core repo is
> that it means we do not have to endorse a particular implementation, and
> they can compete with each other on their merits.  I would very much
> hate to see Felipe's remote-hg project wither and die just because
> another implementation becomes the de facto standard by being included
> in git.git. It's a proven tool, and this new thing is not yet.

Note that I'm only talking about an hypothetical long term goal. If
there's not even a slim chance that this may end up in git core, or in
the git.git repo, I'm not sure it's worth writing the tool in C at all,
considering the burden for users. IOW, I'm only trying to assess if I
should follow my temptation or not. But I can probably reassess after I
actually get my prototype to do more than it does now. But maybe there
are ways to make it work for users outside of git.git even if it's in C.
I don't know.

> It's a shame that both squat on the name "remote-hg", because it makes
> it difficult to tell the two apart. But of course that is the only way
> to make "git clone hg::..." work. Maybe we need a layer of indirection?
> :)

Yeah, that's an unfortunate consequence of how remote helpers work.
There are already two different git-remote-hgs (there's felipe's, and
there's another one using hg-git under the hood) that I know of. I'm
adding a third one. For what it's worth, none of the existing one is
satisfying on repos the size of Mozilla's, and apparently noone at
Mozilla uses them because of that. Add to that the disk space
inefficiency of actually keeping a copy of the mercurial repo locally.
The existing tools can likely be improved to scale better, but that
wouldn't change the disk space problem.

Mike

Re: Announcing a new (prototype) git-remote-hg tool

[Philip Oakley]

From: "Mike Hommey" <mh@glandium.org>
> Hi,
>
> I've (re)started work on a longstanding idea of mine of having a git 
> tool
> talking the mercurial wire protocol directly. I'm now at a stage where
> the tool can clone and pull/fetch from mercurial.
>
> As it is a prototype, there are many things that it doesn't handle 
> (like
> named branches, bookmarks, phases, obsolescence markers), but it
> currently transposes a complete mercurial dag to git and maintains
> metadata about the original mercurial data.
>
> Code on https://github.com/glandium/git-remote-hg
>
> It doesn't support push, but support for that should come in the 
> coming
> weeks.
>
> More background on http://glandium.org/blog/?p=3382
>
> This is meant to be a prototype, and will stay that way for now.
> It's a validation that this can actually work. Now that I have pull
> support I know I can make it push.
> I'm currently evaluating what the final tool would look like. I'm 
> *very*
> tempted to implement it in C, based on core git code, because there 
> are
> many things that this helper does that would be so much easier to do
> with direct access to git's guts. And that wouldn't require more
> dependencies than git currently has: it would "just" need curl and 
> ssh,
> and git already uses both.
>
> If I were to go in that direction, would you consider integrating it
> in git core? If not, would you rather see git helpers to make this
> git-remote-hg helper more efficient?
>
You may also be interested in 
https://felipec.wordpress.com/2013/08/26/whats-new-in-git-v1-8-4-remote-hg/ 
and https://github.com/felipec/git-remote-hg.

Though Filipe's unique work style hasn't found favour locally.

see also https://github.com/buchuki/gitifyhg/wiki/List-of-git-hg-bridges

Philip 

Announcing git-cinnabar 0.1.0 (Was: Announcing a new (prototype) git-remote-hg tool)

[Mike Hommey]

Hi,

Cinnabar is the common natural form in which mercury can be found on
Earth. It contains mercury sulfide and its powder is used to make the
vermillion pigment.

What does that have to do with git?

Hint: mercury.

Git-cinnabar is a git remote helper to interact with mercurial
repositories. It allows to clone, pull and push from/to mercurial remote
repositories, using git.

If you've already seen the original thread this message is spawned from,
this is the same tool, with a new name, and two months worth of
additional work and testing.

Where git-cinnabar stands out compared to other similar tools is that it
doesn't use a local mercurial clone under the hood (unlike all the
existing other such tools), and is close to an order of magnitude faster
to clone a repository like http://hg.mozilla.org/mozilla-central than
the git-remote-hg tool that used to be shipped in contrib/.

I won't claim it is exempt of problems and limitations, which is why it's
not a 1.0. I'm however confident enough with its state to make the
first "official" release.

Code on https://github.com/glandium/git-cinnabar

Cheers,

Mike