Another way to compare tools: is it possible to transfer full history?

Another way to compare tools: is it possible to transfer full history?

[Tuomo]

I have seen lots of comparisons between source control tools, 
but have not found a comparison that would explain the fundamental differences 
and similarities in a way that would really let me choose. 
So I decided to try a new approach: if one tries to transfer the full history 
of an application or a larger product/project, which features can I rely on 
finding in any decently recent tool?

Let's start from Git vs. Mercurial: is it possible to move the whole history 
of an application (with or without submodules) from Git to Mercurial? 
From Mercurial to Git? 
If it is not always possible, what is the feature that might completely 
prevent the whole attempt? If partial transfer is possible, what information 
might be missing in the result?

I am not interested in whether it is easy or fast in practice. 
I am only interested in the concepts and finding equivalent classes 
for the tools or at least their core concepts.

Re: Another way to compare tools: is it possible to transfer full history?

[Tomas Carnecky]

On 9/28/10 3:44 PM, Tuomo wrote:
> I have seen lots of comparisons between source control tools, 
> but have not found a comparison that would explain the fundamental differences 
> and similarities in a way that would really let me choose. 
> So I decided to try a new approach: if one tries to transfer the full history 
> of an application or a larger product/project, which features can I rely on 
> finding in any decently recent tool?
> 
> Let's start from Git vs. Mercurial: is it possible to move the whole history 
> of an application (with or without submodules) from Git to Mercurial? 
> From Mercurial to Git? 

fast-import/export (man git-fast-export/import) seems to be the future.
Git provides excellent support for it and other SCMs are adopting it as
well. And then there are custom written conversion tools, just take a
look at [1] to see which ones are available.

(shameless plug: just this weekend I started collecting the various fast
import/export tools and made a webpage about it:
http://caurea.org/fast-export-import/. It's far from complete though.
And if you know any tools that perform better than those I've listed,
I'd be glade to update the page).

> If it is not always possible, what is the feature that might completely 
> prevent the whole attempt? If partial transfer is possible, what information 
> might be missing in the result?

Not all SCMs have the same features. Subversion for example doesn't have
real branches, tags nor merges (in the same sense that Git does). And
even in distributed version control systems there can be differences.
Git history can't be mapped 1:1 to Mercurial (octopus merges come to
mind). Some of these things can be reasonably emulated, some can't and
you loose that information.

tom

[1]
https://git.wiki.kernel.org/index.php/InterfacesFrontendsAndTools#Interaction_with_other_Revision_Control_Systems

Re: Another way to compare tools: is it possible to transfer full history?

[Michael Haggerty]

On 09/28/2010 04:53 PM, Tomas Carnecky wrote:
> (shameless plug: just this weekend I started collecting the various fast
> import/export tools and made a webpage about it:
> http://caurea.org/fast-export-import/. It's far from complete though.
> And if you know any tools that perform better than those I've listed,
> I'd be glade to update the page).

cvs2svn [1] (in particular the cvs2git [2] and cvs2bzr [3] variants,
which are part of the same project) can convert from CVS to fast-import
format.  It is mature and actively maintained [4], has a lot of features
[5], help is available, and hopefully you agree that the documentation
[6,7] is well-written.

Michael

[1] http://cvs2svn.tigris.org/
[2] http://cvs2svn.tigris.org/cvs2git.html
[3] http://cvs2svn.tigris.org/cvs2bzr.html
[4] https://www.ohloh.net/p/cvs2svn
[5] http://cvs2svn.tigris.org/features.html
[6] http://cvs2svn.tigris.org/cvs2svn.html
[7] http://cvs2svn.tigris.org/faq.html

Re: Another way to compare tools: is it possible to transfer full history?

[Jonathan Nieder]

Tomas Carnecky wrote:

> (shameless plug: just this weekend I started collecting the various fast
> import/export tools and made a webpage about it:
> http://caurea.org/fast-export-import/.

This is awesome --- thank you!

In development:

svn-fe (contrib/svn-fe in Git) converts from an svn dump file to a
fast-import stream. Stale webpage:
http://barrbrain.github.com/svn-dump-fast-export/

remote-hg (see git_remote_helpers/hg/ in
http://github.com/SRabbelier/git) contains a partial hg fast-export
implementation in exporter.py. I don't know how it compares to Rocco
Rutte's exporter.

Not sure if these belong on your list yet, but thought you might like
to know about them. :)

Re: Another way to compare tools: is it possible to transfer full history?

[Sverre Rabbelier]

Heya,

On Tue, Sep 28, 2010 at 22:48, Jonathan Nieder <jrnieder@gmail.com> wrote:
> remote-hg (see git_remote_helpers/hg/ in
> http://github.com/SRabbelier/git) contains a partial hg fast-export
> implementation in exporter.py. I don't know how it compares to Rocco
> Rutte's exporter.

I wrote the exporter from scratch, I know that it works, but I haven't
compared performance against other implementations (such as Rocco
Rutte's)

The accompanying fast-import is based off the bzr one. I'll use the
new python fast-export library that's on pypi in the next iteration
though, so only the part that hooks into hg is interesting.

-- 
Cheers,

Sverre Rabbelier

Re: Another way to compare tools: is it possible to transfer full history?

[Tuomo]

Tomas Carnecky <tom <at> dbservice.com> writes:

> fast-import/export (man git-fast-export/import) seems to be the future.
> Git provides excellent support for it and other SCMs are adopting it as
> well. And then there are custom written conversion tools, just take a
> look at [1] to see which ones are available.
> 
> (shameless plug: just this weekend I started collecting the various fast
> import/export tools and made a webpage about it:
> http://caurea.org/fast-export-import/. It's far from complete though.
> And if you know any tools that perform better than those I've listed,
> I'd be glade to update the page).

Last time I was looking for conversions between source control tools, I could
not find any (but my attempt may have been a bit limited). I am very happy to
see that there is now not only a plethora of point-to-point conversion tools,
but a common exchange format. It means that the field of source control is
finally maturing, finding common concepts that most agree on.

However, the scanty documentation for the tools does not answer my question.
I'll try to formulate it again. You can move history from SCCS to RCS without
losing anything, but you cannot move from RCS to SCCS, because SCCS does not
handle sub-branches. You can move from RCS to CVS, but since RCS does not record
evolutionary relationships between tags, the result does not record the history
in the manner we'd expect in CVS. You can provide additional data to the
conversion, but that additional data cannot be automatically deduced by any
general algorithm. You can move stuff from CVS to RCS, but you lose the
evolution of the whole configuration. So, SCCS, RCS and CVS all belong to a
different class. Only the latest tools have enough in common that one can find
enough of information to make a full transfer of history without loss of data
that can be deduced by automatic means if a back-conversion is desired.

Which tools belong to the same class with Git? Strictly speaking, I am talking
about conversions that do not require us to inject any additional data,
conversions that are fully automatic. The page http://wiki.darcs.net/DarcsGit
mentions that Darcs->Git->Darcs roundtrip loses no information (but also notes
that the tool is nowadays broken and needs fixing), but the same is not true for
the other direction. That is the kind of information I am looking for.

Is it possible to make a round-trip Mercurial->Git->Mercurial or
Git->Mercurial->Git without loss of any information? I would expect that
Mercurial->Git->Mercurial might produce some differences if files have been
renamed or moved between directories, but other than that?

What particularly interests me is how the conversion handles unnamed Mercurial
branches? I am asking this because at work, I had to ponder once if it would be
possible to transfer histories from Synergy (ex Continuus) to some other tool,
and found it very difficult to imagine how to create named branches from the
version DAGs Synergy uses. You can never be sure if a new version is a successor
of its predecessor on the same branch or the first version on a sub.branch,
because Synergy doesn't treat them any differently. users often try to organize
the branches in ways compatible with other tools, but since Synergy has no way
of enforcing any of these methods, there is no guarantee of consistency. The
worst-case scenario is that every single version is its own branch. So, I really
would like to know how the unnamed branches from Mercurial are transferred to
named branches in Git?

Re: Another way to compare tools: is it possible to transfer full history?

[Andreas Ericsson]

On 09/29/2010 01:03 PM, Tuomo wrote:
> Tomas Carnecky<tom<at>  dbservice.com>  writes:
> 
> Which tools belong to the same class with Git?

All repositories from pure distributed version control systems can be
transformed into git repositories and git repos can be transformed into
a repository of a pure distributed version control system, but not all
repositories can do so without information loss since all dvcs systems
store and represent things different and have (slightly) different
capabilities.

Octopus merges have been mentioned. I'm sure there are other things,
such as the three different tag-types we have in git, that can't be
properly represented by other scm systems.

Exactly which those are and how it affects conversions from one
system to another is something you seem to want to learn without
actually doing the job of finding it out, and I doubt anyone on this
list knows every detail you're looking for (although the complete
information you're after might be available in scattered form among
the population on this list).

> 
> Is it possible to make a round-trip Mercurial->Git->Mercurial or
> Git->Mercurial->Git without loss of any information?

That depends. If the git repository has no octopus merges and no tags
of a type that can't be represented in mercurial, I believe it should
be possible.

Try and find out.

> I would expect that
> Mercurial->Git->Mercurial might produce some differences if files have been
> renamed or moved between directories, but other than that?
> 

Possibly. Again though; Try and find out.

> What particularly interests me is how the conversion handles unnamed Mercurial
> branches?

Probably as detached heads. There's nothing special with having commits with no
refs attached to them in git.

> I am asking this because at work, I had to ponder once if it would be
> possible to transfer histories from Synergy (ex Continuus) to some other tool,
> and found it very difficult to imagine how to create named branches from the
> version DAGs Synergy uses. You can never be sure if a new version is a successor
> of its predecessor on the same branch or the first version on a sub.branch,
> because Synergy doesn't treat them any differently. users often try to organize
> the branches in ways compatible with other tools, but since Synergy has no way
> of enforcing any of these methods, there is no guarantee of consistency. The
> worst-case scenario is that every single version is its own branch. So, I really
> would like to know how the unnamed branches from Mercurial are transferred to
> named branches in Git?
> 

So now we get to a proper use-case. You want to convert a Synergy repository to
something else, and you've been looking at mercurial and git.

So let me ask you a question; What have you done so far to find out the answer
to the questions you're looking for, apart from asking here how a theoretical
scenario would pan out?

-- 
Andreas Ericsson                   andreas.ericsson@op5.se
OP5 AB                             www.op5.se
Tel: +46 8-230225                  Fax: +46 8-230231

Considering the successes of the wars on alcohol, poverty, drugs and
terror, I think we should give some serious thought to declaring war
on peace.

Re: Another way to compare tools: is it possible to transfer full history?

[Tuomo]

Andreas Ericsson <ae <at> op5.se> writes:

> 
> So now we get to a proper use-case. You want to convert a Synergy repository to
> something else, and you've been looking at mercurial and git.

Nah, I am not working with Synergy. I referred to an earlier encounter with it.

Right now, I am stuck with ClearCase. Most of the projects have been using base 
ClearCase, not UCM, and it means that there is no realistic way to transfer the 
histories anywhere. It can be tried, sure, but who would really want to do it?
What I am looking for is tools that I can recommend to the projects, tools 
that are available to them, and will not pose a threat in being non-convertible, 
tools that would allow the projects to later make another decision, and not get 
stuck with the tool they choose now.

> So let me ask you a question; What have you done so far to find out the answer
> to the questions you're looking for, apart from asking here how a theoretical
> scenario would pan out?

Fair enough. I am looking for documentation, and advice on how to find worthy 
sources, before delving into trying out something for no good reason. 
But, at the same time, I wish to have a look-see on what the general status 
is right now. If there is no summary of it available, then apparently I have to 
make it myself. But I would hate to find out afterward that someone had in fact 
done it already. Tomas Carnecky's list is a good start.

Re: Another way to compare tools: is it possible to transfer full history?

[Andreas Ericsson]

On 09/29/2010 03:03 PM, Tuomo wrote:
> Andreas Ericsson<ae<at>  op5.se>  writes:
> 
>>
>> So now we get to a proper use-case. You want to convert a Synergy repository to
>> something else, and you've been looking at mercurial and git.
> 
> Nah, I am not working with Synergy. I referred to an earlier encounter with it.
> 
> Right now, I am stuck with ClearCase. Most of the projects have been using base
> ClearCase, not UCM, and it means that there is no realistic way to transfer the
> histories anywhere. It can be tried, sure, but who would really want to do it?
> What I am looking for is tools that I can recommend to the projects, tools
> that are available to them, and will not pose a threat in being non-convertible,
> tools that would allow the projects to later make another decision, and not get
> stuck with the tool they choose now.
> 

In general, it's easier if they decide to change to a distributed version control
system such as git, mercurial or bazaar, since those generally have a much better
data model than centralized systems where you can get away with ugly workarounds
much easier.

Workable conversions are possible between very nearly every scm out there and
whatever you want. It's not necessarily possible to convert back in a way that
makes it look as if you'd used the original scm all the time, but it's (almost)
always possible to create a new repository that is obviously functional and
contains all or most of the relevant history. Such conversions generally take
quite a while for projects with a large history though, so it's not something
you'd ponder doing twice a week.

Most larger projects that have switched have done so with the intention of
using the switched-to system for at least the foreseeable future and have thus
done proper research into what that system has to offer and then made the
decision based on featureset, data model and usability of the available
systems. If you make the decision based on how easy it is to switch away
from the system you choose, you're bound to end up having to do just that
sooner or later.

Either way, very nearly every project in the world who in the past 3 years
have gone looking for a better version control system has gone with one
of the two major distributed systems, git and mercurial. Conversion between
those two is relatively quick and painless and produces working repositories
in both directions, even if git -> hg -> git doesn't necessarily produce a
repository identical to the one you started out with.

>> So let me ask you a question; What have you done so far to find out the answer
>> to the questions you're looking for, apart from asking here how a theoretical
>> scenario would pan out?
> 
> Fair enough. I am looking for documentation, and advice on how to find worthy
> sources, before delving into trying out something for no good reason.
> But, at the same time, I wish to have a look-see on what the general status
> is right now. If there is no summary of it available, then apparently I have to
> make it myself. But I would hate to find out afterward that someone had in fact
> done it already. Tomas Carnecky's list is a good start.
> 

You're looking at the wrong criteria for switching vcs. If the main goal is to
be able to convert the target repository back into the original one, you're up
for disappointment in what you'll find. None of the conversion tools destroy
the original repository though, so you can experiment with any one you like.

Some general truths that might aid you though:
git has the best fast-{import,export} support. Not surprising since the format
was engineered by the brilliant minds we're fortunate to have on the git list.

Conversion between git and other distributed version control systems produce
working repositories in a quick and painfree fashion.

It is usually impossible to convert from one repository format to the other
and back again in such a fashion that the resulting repository is identical
to the one you started with.

It is almost always possible to create a working repository of any kind from
a repository type that has a fast-export tool.

Writing a fast-import tool is not exactly rocket science, although it could
get timeconsuming if the target vcs is limited in capabilities and many
workarounds are necessary.

Most people don't switch to a vcs with fewer capabilities than the one they're
already using, so the previous point is mostly academic.

-- 
Andreas Ericsson                   andreas.ericsson@op5.se
OP5 AB                             www.op5.se
Tel: +46 8-230225                  Fax: +46 8-230231

Considering the successes of the wars on alcohol, poverty, drugs and
terror, I think we should give some serious thought to declaring war
on peace.

Re: Another way to compare tools: is it possible to transfer full history?

[Tuomo]

Andreas Ericsson <ae <at> op5.se> writes:

> You're looking at the wrong criteria for switching vcs.

All right, since this forum is for those who have to have a very concrete 
problem at hands, I am clearly in the wrong company.

What forum would you suggest for more philosophical/academic ponderings? 
The old comp.softare.config-mgmt newsgroup is not available to me (can't 
use newsreader, and it's not on gmane), and it's dead anyway, full of 
commercial bs instead of discussion. Where has the general discussion 
moved?

Re: Another way to compare tools: is it possible to transfer full history?

[Bruce Stephens]

Tuomo <tuo.tie@gmail.com> writes:

[...]

> Where has the general discussion moved?

Dunno.  There's a (extremely low-volume) revctrl mailing list with a
marginally more active IRC channel.  Maybe that's it, or maybe people
are using blogs?