[RFC] mtn to git conversion script

[RFC] mtn to git conversion script

[Felipe Contreras]

Hi,

I developed a script that converts a monotone repository into a git
one (exact clone), I want to contribute it so everybody can use it.

However, I might have not done it correctly.

This is the gist of the script:

mtn update --revision #{@id} --reallyquiet
git ls-files --modified --others --exclude-standard -z | git
update-index --add --remove -z --stdin
git write-tree
git write-raw < /tmp/commit.txt
git update-ref refs/mtn/#{@id} #{@git_id}

branches.each do |e|
    git update-ref refs/heads/#{e} #{@git_id}
end

I wrote "git write-raw" which takes the commit text as is, and puts it
into the repository.

I've read about 'fast-import' but I'm not sure if it would be more
efficient, because you would have to parse the output of different mtn
tools.

What do you think? Does it makes sense to have a 'write-raw' command?
Or should I somehow use 'fast-import'?

Best regards.

-- 
Felipe Contreras

Re: [RFC] mtn to git conversion script

[Miklos Vajna]

[-- Attachment #1: Type: text/plain, Size: 415 bytes --]

On Sun, Aug 24, 2008 at 12:18:50PM +0300, Felipe Contreras <felipe.contreras@gmail.com> wrote:
> What do you think? Does it makes sense to have a 'write-raw' command?
> Or should I somehow use 'fast-import'?

Yes, you should. ;-)

The syntax of it is not so hard, see for example 'git fast-export
HEAD~2..' on a git repo and you'll see.

This should help a lot if you are like me, who likes to learn from
examples.

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: [RFC] mtn to git conversion script

[Johannes Schindelin]

Hi,

On Sun, 24 Aug 2008, Miklos Vajna wrote:

> On Sun, Aug 24, 2008 at 12:18:50PM +0300, Felipe Contreras <felipe.contreras@gmail.com> wrote:
> > What do you think? Does it makes sense to have a 'write-raw' command? 
> > Or should I somehow use 'fast-import'?
> 
> Yes, you should. ;-)
> 
> The syntax of it is not so hard, see for example 'git fast-export
> HEAD~2..' on a git repo and you'll see.
> 
> This should help a lot if you are like me, who likes to learn from
> examples.

Heh.  I am glad you like fast-export.  To be honest, I never intended to 
use fast-export for anything else than as an example how to drive 
fast-import... :-)

Ciao,
Dscho

Re: [RFC] mtn to git conversion script

[Felipe Contreras]

On Sun, Aug 24, 2008 at 4:14 PM, Miklos Vajna <vmiklos@frugalware.org> wrote:
> On Sun, Aug 24, 2008 at 12:18:50PM +0300, Felipe Contreras <felipe.contreras@gmail.com> wrote:
>> What do you think? Does it makes sense to have a 'write-raw' command?
>> Or should I somehow use 'fast-import'?
>
> Yes, you should. ;-)
>
> The syntax of it is not so hard, see for example 'git fast-export
> HEAD~2..' on a git repo and you'll see.
>
> This should help a lot if you are like me, who likes to learn from
> examples.

Is it possible to create a fast-import from the index? I realize this
is not the best thing to do, but for now I would like to do that.

Best regards.

-- 
Felipe Contreras

Re: [RFC] mtn to git conversion script

[Miklos Vajna]

[-- Attachment #1: Type: text/plain, Size: 359 bytes --]

On Sun, Aug 24, 2008 at 08:19:23PM +0200, Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:
> Heh.  I am glad you like fast-export.  To be honest, I never intended to 
> use fast-export for anything else than as an example how to drive 
> fast-import... :-)

It's much more, git-bzr's bi-directional operation would be impossible
without it. ;-)

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: [RFC] mtn to git conversion script

[Shawn O. Pearce]

Felipe Contreras <felipe.contreras@gmail.com> wrote:
> On Sun, Aug 24, 2008 at 4:14 PM, Miklos Vajna <vmiklos@frugalware.org> wrote:
> > On Sun, Aug 24, 2008 at 12:18:50PM +0300, Felipe Contreras <felipe.contreras@gmail.com> wrote:
> >> What do you think? Does it makes sense to have a 'write-raw' command?
> >> Or should I somehow use 'fast-import'?
> >
> > Yes, you should. ;-)
> >
> > The syntax of it is not so hard, see for example 'git fast-export
> > HEAD~2..' on a git repo and you'll see.
> >
> > This should help a lot if you are like me, who likes to learn from
> > examples.
> 
> Is it possible to create a fast-import from the index? I realize this
> is not the best thing to do, but for now I would like to do that.

No, fast-import uses its own internal structure and avoids the
index file.

Also, look at `git-hash-objects -w` as a replacement for your
git-write-raw tool if you aren't going to use git-fast-import.

-- 
Shawn.

Re: [RFC] mtn to git conversion script

[Felipe Contreras]

On Mon, Aug 25, 2008 at 1:46 AM, Shawn O. Pearce <spearce@spearce.org> wrote:
> Felipe Contreras <felipe.contreras@gmail.com> wrote:
>> On Sun, Aug 24, 2008 at 4:14 PM, Miklos Vajna <vmiklos@frugalware.org> wrote:
>> > On Sun, Aug 24, 2008 at 12:18:50PM +0300, Felipe Contreras <felipe.contreras@gmail.com> wrote:
>> >> What do you think? Does it makes sense to have a 'write-raw' command?
>> >> Or should I somehow use 'fast-import'?
>> >
>> > Yes, you should. ;-)
>> >
>> > The syntax of it is not so hard, see for example 'git fast-export
>> > HEAD~2..' on a git repo and you'll see.
>> >
>> > This should help a lot if you are like me, who likes to learn from
>> > examples.
>>
>> Is it possible to create a fast-import from the index? I realize this
>> is not the best thing to do, but for now I would like to do that.
>
> No, fast-import uses its own internal structure and avoids the
> index file.

Yeah, I knew that, but wanted to just replace the 'write-raw' command.
To avoid doing unnecessary changes.

> Also, look at `git-hash-objects -w` as a replacement for your
> git-write-raw tool if you aren't going to use git-fast-import.

Awesome, but I just did it properly :)

A few comments regarding fast-import:

Why the distinction between 'from' and 'merge'? Doesn't it make more
sense to use 'parent' for both?

I'm doing: commit refs/mtn/d137c7046bae7e4a0144fee82bfce8061f61e3b3

So I was expecing this to work:
from mtn/d137c7046bae7e4a0144fee82bfce8061f61e3b3

But it didn't, probably because the commit hasn't actually been
committed. Wouldn't it make sense to store it as a temporal commit so
my script doesn't have to deal with that?

Anyway, very nice tool. It's going much faster (1h) compared to before (1 day).

Best regards.

-- 
Felipe Contreras

Re: [RFC] mtn to git conversion script

[Brian Downing]

[-- Attachment #1: Type: text/plain, Size: 1740 bytes --]

On Sun, Aug 24, 2008 at 12:18:50PM +0300, Felipe Contreras wrote:
> I developed a script that converts a monotone repository into a git
> one (exact clone), I want to contribute it so everybody can use it.
> 
> This is the gist of the script:
> 
> mtn update --revision #{@id} --reallyquiet
> git ls-files --modified --others --exclude-standard -z | git
> update-index --add --remove -z --stdin
> git write-tree
> git write-raw < /tmp/commit.txt
> git update-ref refs/mtn/#{@id} #{@git_id}
> 
> branches.each do |e|
>     git update-ref refs/heads/#{e} #{@git_id}
> end

You definitely want to use fast-import, but you probably want to do
something a lot closer to fast-export for monotone (read: use its
automate stdio interface and avoid expensive calls).

Here's a simple monotone to git converter I wrote.  You'll need the
Monotone::AutomateStdio perl module to use it (which I think I got it
from monotone's net.venge.monotone.contrib.lib.automate-stdio branch).
It is very fast; it can convert the OpenEmbedded repo in something like
5-10 minutes on my machine.

Note that for monotone export to go fast you absolutely /must/ avoid the
get_manifest operation.  In my converter I use the revision information
directly.  Getting the renames right with this is a little tricky; IIRC,
the ordering that works is:

* Rename all renamed files, innermost files first, to temporary names.
* Delete all deleted files, innermost first.
* Rename all temporary names to permanent names, outermost first.
* Add all new/modified files.

Conveniently, all of the above can be done by directly emitting
fast-import commands, so you don't have to keep track of trees directly.
(With one exception, which I'll elaborate on in a different email.)

-bcd

[-- Attachment #2: mtn-to-git.pl --]
[-- Type: text/x-perl, Size: 4759 bytes --]

#!/usr/bin/perl
# Copyright (C) 2007-2008  Brian Downing
# This program is licensed under version 2 of the GNU GPL.

use strict;
use Monotone::AutomateStdio;
use Date::Parse;

my $D = 0;

my $m = Monotone::AutomateStdio->new($ARGV[0]);

my $revlist = [];
$m->graph($revlist);
my $sorted = [];

for my $rev (@$revlist) {
    push(@$sorted, $rev->{revision_id});
}

my $leaves = [];
$m->leaves($leaves);

$m->toposort($sorted, @$sorted);

my $marks = {};
my $mark = 1;
my $c = 0;

sub quote_file {
    $_ = shift;
    return $_;
    s/\\/\\\\/g;
    s/\n/\\n/g;
    s/"/\\"/g;
    return qq("$_");
}

sub lprint {
    my $fh = shift;
    print @_ if $D;
    print $fh @_;
}

sub lprintf {
    my $fh = shift;
    printf @_ if $D;
    printf $fh @_;
}

my $tmptag = "624d893e-ae1a-42d8-90a9-926a6ceffae8";

open my $fi, '|-', 'git-fast-import --export-marks=file';
for my $rev (@$sorted) {
    my ($time, $author, $msg) = ("0", "__UNKNOWN__", "__UNKNOWN__");
    my @certs;
    my @branches;
    $m->certs(\@certs, $rev);
    for my $cert (@certs) {
        my ($n, $v) = ($cert->{name}, $cert->{value});
        $author = $v if ($n eq 'author');
        $time = $v if ($n eq 'date');
        $msg = $v if ($n eq 'changelog');
        push(@branches, $v) if ($n eq 'branch');
    }
    my $email = $author;
    $msg .= "\nmtn-revision: $rev\n";
    for my $b (sort @branches) {
        $msg .= "mtn-branch: $b\n";
    }
    $time = str2time($time, 'UTC');
    my $mfest = [];
    $m->get_revision($mfest, $rev);
    my $orcount = 0;
    my $add_files = {};
    my $add_dirs = {};
    my $delete_files = {};
    my $from_tmpnames = {};
    my $to_tmpnames = {};
    my $curtmp = 0;
    my @parents;
    for my $e (@$mfest) {
        if ($e->{type} eq 'old_revision') {
            push(@parents, $e->{revision_id});
            ++$orcount;
        } 
        next if $orcount > 1;
        if ($e->{type} eq 'add_file' || $e->{type} eq 'patch') {
            my $id = $e->{file_id} || $e->{to_file_id};
            $add_files->{$e->{name}} = $id;
            unless ($marks->{$id}) {
                my $data;
                $m->get_file(\$data, $id);
                print "new file $id\n" if $D;
                print $fi "blob\n";
                my $len = length($data);
                print $fi "mark :$mark\n";
                $marks->{$id} = $mark++;
                print $fi "data $len\n$data\n";
            }
        } elsif ($e->{type} eq 'add_dir') {
            $add_dirs->{$e->{name}} = 1;
        } elsif ($e->{type} eq 'delete') {
            $delete_files->{$e->{name}} = 1;
        } elsif ($e->{type} eq 'rename') {
            $curtmp++;
            $from_tmpnames->{$e->{from_name}} = "__tmp_${tmptag}_$curtmp";
            $to_tmpnames->{$e->{to_name}} = "__tmp_${tmptag}_$curtmp";
        }
    }
    printf("rev $rev (%d/%d, %.2f%)\n",
           ++$c, scalar(@$sorted), 100*$c/scalar(@$sorted));
    print $fi "reset refs/import\n" unless @parents;
    lprint $fi, "commit refs/import\n";
    print $fi "mark :$mark\n";
    $marks->{$rev} = $mark++;
    if ($author =~ m(\s*(.*?\S)\s*<(.*)>\s*)) {
        $author = $1;
        $email = $2;
    }
    $author =~ s/[<>]/_/g;
    $email =~ s/[<>]/_/g;
    $author =~ s/@.*//;
    print $fi "committer $author <$email> $time +0000\n";
    my $len = length($msg);
    print $fi "data $len\n$msg\n";
    my $from = "from";
    for my $p (@parents) {
        lprint $fi, "$from :$marks->{$p}\n";
        $from = "merge";
    }
    for my $f (sort { length($b) <=> length ($a) } keys %$from_tmpnames) {
        lprintf($fi, "R %s %s\n",
                quote_file($f), quote_file($from_tmpnames->{$f}));
    }
    for my $f (sort { length($b) <=> length ($a) } keys %$delete_files) {
        lprintf($fi, "D %s\n", quote_file($f));
    }
    for my $f (sort { length($a) <=> length ($b) } keys %$to_tmpnames) {
        lprintf($fi, "R %s %s\n",
                quote_file($to_tmpnames->{$f}), quote_file($f));
    }
    for my $f (keys %$add_files) {
        lprintf($fi, "M 0644 :%s %s\n",
                $marks->{$add_files->{$f}}, quote_file($f));
    }
    for my $f (keys %$add_dirs) {
        $f .= "/" if $f;
        lprintf($fi, "M 0644 inline %s\n", quote_file("$f.gitignore"));
        lprint($fi, "data 0\n\n");
    }
    print $fi "\n";
}

my $branches = {};
for my $rev (@$leaves) {
    my $branch;
    my @certs;
    $m->certs(\@certs, $rev);
    for my $cert (@certs) {
        my ($n, $v) = ($cert->{name}, $cert->{value});
        $branch = $v if ($n eq 'branch');
    }
    my $r = $branches->{$branch};
    $branches->{$branch}--;
    if ($marks->{$rev}) {
        print $fi "reset refs/heads/$branch$r\n";
        print $fi "from :$marks->{$rev}\n\n";
    }
}

close $fi;

Re: [RFC] mtn to git conversion script

[Brian Downing]

On Mon, Aug 25, 2008 at 11:35:31AM -0500, Brian Downing wrote:
> Note that for monotone export to go fast you absolutely /must/ avoid the
> get_manifest operation.  In my converter I use the revision information
> directly.  Getting the renames right with this is a little tricky; IIRC,
> the ordering that works is:
> 
> * Rename all renamed files, innermost files first, to temporary names.
> * Delete all deleted files, innermost first.
> * Rename all temporary names to permanent names, outermost first.
> * Add all new/modified files.
> 
> Conveniently, all of the above can be done by directly emitting
> fast-import commands, so you don't have to keep track of trees directly.
> (With one exception, which I'll elaborate on in a different email.)

The exception is one commit in monotone's repository.  There was
actually a commit that did:

    rename '/' '/something'
    add '/other'

Monotone can apparently handle that, but git fast-import cannot, last I
checked.  One would have to "know" what all the files were and recreate
them by hand, which was what fast-import's move/copy commands were
supposed to avoid.

Obviously this use case is not too important to me, as patches have not
been forthcoming to fix this, but I figured I'd mention it in case it's
important to somebody else.

-bcd

Re: [RFC] mtn to git conversion script

[Felipe Contreras]

On Mon, Aug 25, 2008 at 7:35 PM, Brian Downing <bdowning@lavos.net> wrote:
> On Sun, Aug 24, 2008 at 12:18:50PM +0300, Felipe Contreras wrote:
>> I developed a script that converts a monotone repository into a git
>> one (exact clone), I want to contribute it so everybody can use it.
>>
>> This is the gist of the script:
>>
>> mtn update --revision #{@id} --reallyquiet
>> git ls-files --modified --others --exclude-standard -z | git
>> update-index --add --remove -z --stdin
>> git write-tree
>> git write-raw < /tmp/commit.txt
>> git update-ref refs/mtn/#{@id} #{@git_id}
>>
>> branches.each do |e|
>>     git update-ref refs/heads/#{e} #{@git_id}
>> end
>
> You definitely want to use fast-import, but you probably want to do
> something a lot closer to fast-export for monotone (read: use its
> automate stdio interface and avoid expensive calls).
>
> Here's a simple monotone to git converter I wrote.  You'll need the
> Monotone::AutomateStdio perl module to use it (which I think I got it
> from monotone's net.venge.monotone.contrib.lib.automate-stdio branch).
> It is very fast; it can convert the OpenEmbedded repo in something like
> 5-10 minutes on my machine.

Interesting, how many commits?

> Note that for monotone export to go fast you absolutely /must/ avoid the
> get_manifest operation.  In my converter I use the revision information
> directly.  Getting the renames right with this is a little tricky; IIRC,
> the ordering that works is:
>
> * Rename all renamed files, innermost files first, to temporary names.
> * Delete all deleted files, innermost first.
> * Rename all temporary names to permanent names, outermost first.
> * Add all new/modified files.
>
> Conveniently, all of the above can be done by directly emitting
> fast-import commands, so you don't have to keep track of trees directly.
> (With one exception, which I'll elaborate on in a different email.)

I guess I haven't stumbled upon that problem yet =/

Best regards.

-- 
Felipe Contreras

Re: [RFC] mtn to git conversion script

[Brian Downing]

On Mon, Aug 25, 2008 at 11:47:53PM +0300, Felipe Contreras wrote:
> On Mon, Aug 25, 2008 at 7:35 PM, Brian Downing <bdowning@lavos.net> wrote:
> > Here's a simple monotone to git converter I wrote.  You'll need the
> > Monotone::AutomateStdio perl module to use it (which I think I got it
> > from monotone's net.venge.monotone.contrib.lib.automate-stdio branch).
> > It is very fast; it can convert the OpenEmbedded repo in something like
> > 5-10 minutes on my machine.
> 
> Interesting, how many commits?

:; git rev-list --all | wc -l
23498 revisions
:; git ls-tree -r org.openembedded.stable | wc -l
17502 files in HEAD

(Some of those files are .gitignore files, which I create in every
directory to hold open empty ones.)

-bcd

Re: [RFC] mtn to git conversion script

[Anand Kumria]

Hi Felipe,

On Mon, 25 Aug 2008 03:45:11 +0300, Felipe Contreras wrote:

> 
> Anyway, very nice tool. It's going much faster (1h) compared to before
> (1 day).

Will you be submitting this as something for/to contrib?

Thanks,
Anand

Re: [RFC] mtn to git conversion script

[Anand Kumria]

Hi Brian,

On Mon, 25 Aug 2008 11:35:31 -0500, Brian Downing wrote:

[snip - Convenient mtn -> git converter ]

I think you need to add a Signed-off-by: line in order for Junio to be 
able to take this and put into the contrib section.

Thanks,
Anand

Re: [Monotone-devel] Re: [RFC] mtn to git conversion script

[Felipe Contreras]

On Thu, Aug 28, 2008 at 8:57 AM, Anand Kumria <wildfire@progsoc.org> wrote:
>
> Hi Felipe,
>
> On Mon, 25 Aug 2008 03:45:11 +0300, Felipe Contreras wrote:
>
>>
>> Anyway, very nice tool. It's going much faster (1h) compared to before
>> (1 day).
>
> Will you be submitting this as something for/to contrib?

Yes, that's the plan.

However, I still don't have something that creates an exact clone with
fast-import.

Also, I'm trying different ways to see what would be most efficient.
Right now it's a combination of Ruby + C, but once I get it working
I'll post it to the OE mailing lists to see if it works fine for them
too.

Once the design is good enough I might move everything to C.

Best regards.

-- 
Felipe Contreras

Re: [Monotone-devel] Re: [RFC] mtn to git conversion script

[Felipe Contreras]

On Thu, Aug 28, 2008 at 12:03 PM, Felipe Contreras
<felipe.contreras@gmail.com> wrote:
> On Thu, Aug 28, 2008 at 8:57 AM, Anand Kumria <wildfire@progsoc.org> wrote:
>>
>> Hi Felipe,
>>
>> On Mon, 25 Aug 2008 03:45:11 +0300, Felipe Contreras wrote:
>>
>>>
>>> Anyway, very nice tool. It's going much faster (1h) compared to before
>>> (1 day).
>>
>> Will you be submitting this as something for/to contrib?
>
> Yes, that's the plan.
>
> However, I still don't have something that creates an exact clone with
> fast-import.
>
> Also, I'm trying different ways to see what would be most efficient.
> Right now it's a combination of Ruby + C, but once I get it working
> I'll post it to the OE mailing lists to see if it works fine for them
> too.
>
> Once the design is good enough I might move everything to C.
>
> Best regards.

Ok, now the basics seem to be working. So I'm uploading some code if
anyone wants to take a look.

The C code is generating a topologically sorted list of revisions, and
storing the relevant information (certs and parents) separately. This
code is very fast. It's using GLib and sqlite3, so probably the GLib
stuff should be converted to use libgit.
http://gist.github.com/8742

The Ruby code takes that input and generates an output suitable for
fast-import. It would be tedious to port the parsing stuff to C, but
straightforward.
http://gist.github.com/8740

A patch for fast-import is required, I already submitted it.

Comments?

-- 
Felipe Contreras

Re: [Monotone-devel] Re: [RFC] mtn to git conversion script

[Jakub Narebski]

"Felipe Contreras" <felipe.contreras@gmail.com> writes:

> Ok, now the basics seem to be working. So I'm uploading some code if
> anyone wants to take a look.
> 
> The C code is generating a topologically sorted list of revisions, and
> storing the relevant information (certs and parents) separately. This
> code is very fast. It's using GLib and sqlite3, so probably the GLib
> stuff should be converted to use libgit.
> http://gist.github.com/8742
> 
> The Ruby code takes that input and generates an output suitable for
> fast-import. It would be tedious to port the parsing stuff to C, but
> straightforward.
> http://gist.github.com/8740
> 
> A patch for fast-import is required, I already submitted it.
> 
> Comments?

If you feel like it is good enough, could you add information about
this tool to Git Wiki:
  http://git.or.cz/gitwiki/InterfacesFrontendsAndTools
in the "Interaction with other Revision Control Systems" section?

TIA
-- 
Jakub Narebski
Poland
ShadeHawk on #git

Re: [Monotone-devel] Re: [RFC] mtn to git conversion script

[Thomas Moschny]

[-- Attachment #1: Type: text/plain, Size: 757 bytes --]

On Thu, Sep 4, 2008, Felipe Contreras wrote:
> Ok, now the basics seem to be working. So I'm uploading some code if
> anyone wants to take a look.
>
> The C code is generating a topologically sorted list of revisions, and
> storing the relevant information (certs and parents) separately. This
> code is very fast. It's using GLib and sqlite3, so probably the GLib
> stuff should be converted to use libgit.
> http://gist.github.com/8742

You shouldn't access Monotone's sqlite database directly, for various reasons. 
Use the Automation Interface instead, see 
http://www.monotone.ca/docs/Automation.html#Automation. Using 'mtn automate 
stdio', you can feed an arbitrary amount of commands to one single running mtn 
process.

- Thomas


[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: [Monotone-devel] Re: [RFC] mtn to git conversion script

[Felipe Contreras]

On Thu, Sep 4, 2008 at 1:31 PM, Jakub Narebski <jnareb@gmail.com> wrote:
> "Felipe Contreras" <felipe.contreras@gmail.com> writes:
>
>> Ok, now the basics seem to be working. So I'm uploading some code if
>> anyone wants to take a look.
>>
>> The C code is generating a topologically sorted list of revisions, and
>> storing the relevant information (certs and parents) separately. This
>> code is very fast. It's using GLib and sqlite3, so probably the GLib
>> stuff should be converted to use libgit.
>> http://gist.github.com/8742
>>
>> The Ruby code takes that input and generates an output suitable for
>> fast-import. It would be tedious to port the parsing stuff to C, but
>> straightforward.
>> http://gist.github.com/8740
>>
>> A patch for fast-import is required, I already submitted it.
>>
>> Comments?
>
> If you feel like it is good enough, could you add information about
> this tool to Git Wiki:
>  http://git.or.cz/gitwiki/InterfacesFrontendsAndTools
> in the "Interaction with other Revision Control Systems" section?

Not yet.

It still needs to add the branches, tags, and HEAD.

-- 
Felipe Contreras

Re: [Monotone-devel] Re: [RFC] mtn to git conversion script

[Felipe Contreras]

On Thu, Sep 4, 2008 at 1:50 PM, Thomas Moschny <thomas.moschny@gmx.de> wrote:
> On Thu, Sep 4, 2008, Felipe Contreras wrote:
>> Ok, now the basics seem to be working. So I'm uploading some code if
>> anyone wants to take a look.
>>
>> The C code is generating a topologically sorted list of revisions, and
>> storing the relevant information (certs and parents) separately. This
>> code is very fast. It's using GLib and sqlite3, so probably the GLib
>> stuff should be converted to use libgit.
>> http://gist.github.com/8742
>
> You shouldn't access Monotone's sqlite database directly, for various reasons.
> Use the Automation Interface instead, see
> http://www.monotone.ca/docs/Automation.html#Automation. Using 'mtn automate
> stdio', you can feed an arbitrary amount of commands to one single running mtn
> process.

I use mtn stdio when needed, that is, when doing it manually would be
too complicated (get_file). Doing it directly with sqlite3 is *very*
fast, I don't see any reason to not to do it.

Feel free to modify the code for stdio.

-- 
Felipe Contreras

Re: [RFC] mtn to git conversion script

[Juan Jose Comellas]

[-- Attachment #1.1: Type: text/plain, Size: 2689 bytes --]

I made some modifications to the script that converts Monotone repositories
to Git to make it work with what I had. I also added support for renaming
commit authors. To use the modified script just call it passing the name of
the repository file as the first argument. You can add a second optional
argument with the name of the file that holds the authors' names and email
addresses. In this file you should have one line per commit author with the
following format:

Firstname Lastname <email@example.com>

This script still uses the AutomateStdio.pm Perl module that can be found in
the net.venge.monotone.contrib.lib.automate-stdio branch of Monotone's main
repository.

PS. I'm no Perl guru so there might be some bugs lurking in the code I
added. It did work for my repositories, though.


On Mon, Aug 25, 2008 at 2:35 PM, Brian Downing <bdowning@lavos.net> wrote:

> On Sun, Aug 24, 2008 at 12:18:50PM +0300, Felipe Contreras wrote:
> > I developed a script that converts a monotone repository into a git
> > one (exact clone), I want to contribute it so everybody can use it.
> >
> > This is the gist of the script:
> >
> > mtn update --revision #{@id} --reallyquiet
> > git ls-files --modified --others --exclude-standard -z | git
> > update-index --add --remove -z --stdin
> > git write-tree
> > git write-raw < /tmp/commit.txt
> > git update-ref refs/mtn/#{@id} #{@git_id}
> >
> > branches.each do |e|
> >     git update-ref refs/heads/#{e} #{@git_id}
> > end
>
> You definitely want to use fast-import, but you probably want to do
> something a lot closer to fast-export for monotone (read: use its
> automate stdio interface and avoid expensive calls).
>
> Here's a simple monotone to git converter I wrote.  You'll need the
> Monotone::AutomateStdio perl module to use it (which I think I got it
> from monotone's net.venge.monotone.contrib.lib.automate-stdio branch).
> It is very fast; it can convert the OpenEmbedded repo in something like
> 5-10 minutes on my machine.
>
> Note that for monotone export to go fast you absolutely /must/ avoid the
> get_manifest operation.  In my converter I use the revision information
> directly.  Getting the renames right with this is a little tricky; IIRC,
> the ordering that works is:
>
> * Rename all renamed files, innermost files first, to temporary names.
> * Delete all deleted files, innermost first.
> * Rename all temporary names to permanent names, outermost first.
> * Add all new/modified files.
>
> Conveniently, all of the above can be done by directly emitting
> fast-import commands, so you don't have to keep track of trees directly.
> (With one exception, which I'll elaborate on in a different email.)
>
> -bcd
>

[-- Attachment #1.2: Type: text/html, Size: 3263 bytes --]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: mtn-to-git.pl --]
[-- Type: text/x-perl; name=mtn-to-git.pl, Size: 5793 bytes --]

#!/usr/bin/perl
# Copyright (C) 2007-2008  Brian Downing
# This program is licensed under version 2 of the GNU GPL.

use strict;
use Monotone::AutomateStdio;
use Date::Parse;

my $D = 0;

my $m = Monotone::AutomateStdio->new($ARGV[0]);

my $revlist = [];
$m->graph($revlist);
my $sorted = [];

for my $rev (@$revlist) {
    push(@$sorted, $rev->{revision_id});
}

my $leaves = [];
$m->leaves($leaves);

$m->toposort($sorted, @$sorted);

my $marks = {};
my $mark = 1;
my $c = 0;
my %authors = {};

sub quote_file {
    $_ = shift;
    return $_;
    s/\\/\\\\/g;
    s/\n/\\n/g;
    s/"/\\"/g;
    return qq("$_");
}

sub lprint {
    my $fh = shift;
    print @_ if $D;
    print $fh @_;
}

sub lprintf {
    my $fh = shift;
    printf @_ if $D;
    printf $fh @_;
}

sub load_authors {
    my %authors = ();
    my $filename = shift(@_);
    open my $fi, '<', $filename or die "Could not open authors map file $filename\n"; 
    for my $line (<$fi>) {
        if ($line =~ m/(.*) <(.*)>/) {
            if ($2) {
                $authors{$2} = $1;
            }
        }
    }
    return %authors;
}

sub author_name {
    my $email = shift(@_);
    my $name = $authors{$email};
    if ($name) {
        return $name;
    } else {
        if ($email =~ m/(.+)\@.+/) {
            return $1;
        } else {
            return $email;
        }
    }
}

my $tmptag = "624d893e-ae1a-42d8-90a9-926a6ceffae8";

if ($ARGV[1]) {
    %authors = load_authors($ARGV[1]);
} else {
    %authors = {};
}
open my $fi, '|-', 'git-fast-import --export-marks=file';
# open my $fi, '>fast-import.dump';
binmode $fi;
for my $rev (@$sorted) {
    my ($time, $author, $msg) = ("0", "__UNKNOWN__", "__UNKNOWN__");
    my @certs;
    my @branches;
    $m->certs(\@certs, $rev);
    for my $cert (@certs) {
        my ($n, $v) = ($cert->{name}, $cert->{value});
        $author = $v if ($n eq 'author');
        $time = $v if ($n eq 'date');
        $msg = $v if ($n eq 'changelog');
        push(@branches, $v) if ($n eq 'branch');
    }
    my $email = $author;
    $msg .= "\nmtn-revision: $rev\n";
    for my $b (sort @branches) {
        $msg .= "mtn-branch: $b\n";
    }
    $time = str2time($time, 'UTC');
    my $mfest = [];
    $m->get_revision($mfest, $rev);
    my $orcount = 0;
    my $add_files = {};
    my $add_dirs = {};
    my $delete_files = {};
    my $from_tmpnames = {};
    my $to_tmpnames = {};
    my $curtmp = 0;
    my @parents;
    for my $e (@$mfest) {
        if ($e->{type} eq 'old_revision') {
            push(@parents, $e->{revision_id});
            ++$orcount;
        } 
        next if $orcount > 1;
        if ($e->{type} eq 'add_file' || $e->{type} eq 'patch') {
            my $id = $e->{file_id} || $e->{to_file_id};
            $add_files->{$e->{name}} = $id;
            unless ($marks->{$id}) {
                my $data;
                $m->get_file(\$data, $id);
                print "new file $id\n" if $D;
                print $fi "blob\n";
                my $len = length($data);
                print $fi "mark :$mark\n";
                $marks->{$id} = $mark++;
                print $fi "data $len\n$data\n";
                #print $fi "data $len\n";
                #print $fi pack('C', $data);
                #print $fi "\n";
            }
        } elsif ($e->{type} eq 'add_dir') {
            $add_dirs->{$e->{name}} = 1;
        } elsif ($e->{type} eq 'delete') {
            $delete_files->{$e->{name}} = 1;
        } elsif ($e->{type} eq 'rename') {
            $curtmp++;
            $from_tmpnames->{$e->{from_name}} = "__tmp_${tmptag}_$curtmp";
            $to_tmpnames->{$e->{to_name}} = "__tmp_${tmptag}_$curtmp";
        }
    }
    printf("rev $rev (%d/%d, %.2f%)\n",
           ++$c, scalar(@$sorted), 100*$c/scalar(@$sorted));
    print $fi "reset refs/import\n" unless @parents;
    lprint $fi, "commit refs/import\n";
    print $fi "mark :$mark\n";
    $marks->{$rev} = $mark++;
    if ($author =~ m(\s*(.*?\S)\s*<(.*)>\s*)) {
        # $author = $1;
        $email = $2;
    }
    # $author =~ s/[<>]/_/g;
    $email =~ s/[<>]/_/g;
    # $author =~ s/@.*//;
    $author = author_name($email);
    print $fi "committer $author <$email> $time +0000\n";
    my $len = length($msg);
    print $fi "data $len\n$msg\n";
    my $from = "from";
    for my $p (@parents) {
        # lprint $fi, "$from :$marks->{$p}\n";
        my $parent_mark = $marks->{$p};
        if ($parent_mark) {
            lprint $fi, "$from :$parent_mark\n";
            $from = "merge";
        }
    }
    for my $f (sort { length($b) <=> length ($a) } keys %$from_tmpnames) {
        lprintf($fi, "R %s %s\n",
                quote_file($f), quote_file($from_tmpnames->{$f}));
    }
    for my $f (sort { length($b) <=> length ($a) } keys %$delete_files) {
        lprintf($fi, "D %s\n", quote_file($f));
    }
    for my $f (sort { length($a) <=> length ($b) } keys %$to_tmpnames) {
        lprintf($fi, "R %s %s\n",
                quote_file($to_tmpnames->{$f}), quote_file($f));
    }
    for my $f (keys %$add_files) {
        lprintf($fi, "M 0644 :%s %s\n",
                $marks->{$add_files->{$f}}, quote_file($f));
    }
    for my $f (keys %$add_dirs) {
        $f .= "/" if $f;
        lprintf($fi, "M 0644 inline %s\n", quote_file("$f.gitignore"));
        lprint($fi, "data 0\n\n");
    }
    print $fi "\n";
}

my $branches = {};
for my $rev (@$leaves) {
    my $branch;
    my @certs;
    $m->certs(\@certs, $rev);
    for my $cert (@certs) {
        my ($n, $v) = ($cert->{name}, $cert->{value});
        $branch = $v if ($n eq 'branch');
    }
    my $r = $branches->{$branch};
    $branches->{$branch}--;
    if ($marks->{$rev}) {
        print $fi "reset refs/heads/$branch$r\n";
        print $fi "from :$marks->{$rev}\n\n";
    }
}

close $fi;

[-- Attachment #3: Type: text/plain, Size: 158 bytes --]

_______________________________________________
Monotone-devel mailing list
Monotone-devel@nongnu.org
http://lists.nongnu.org/mailman/listinfo/monotone-devel

Re: [RFC] mtn to git conversion script

[Juan Jose Comellas]

[-- Attachment #1: Type: text/plain, Size: 2761 bytes --]

I made some modifications to the script that converts Monotone
repositories to Git to make it work with what I had. I also added
support for renaming commit authors.

To use the modified script just call it passing the name of the
repository file as the first argument. You can add a second optional
argument with the name of the file that holds the authors' names and
email addresses. In this file you should have one line per commit
author with the following format:

Firstname Lastname <email@example.com>

This script still uses the AutomateStdio.pm Perl module that can be
found in the net.venge.monotone.contrib.lib.automate-stdio branch of
Monotone's main repository.

I'm no Perl guru so there might be some bugs lurking in the code I
added. It did work for my repositories, though.

PS. Resending because I mistakenly sent the previous message as HTML mail.


On Mon, Aug 25, 2008 at 2:35 PM, Brian Downing <bdowning@lavos.net> wrote:
>
> On Sun, Aug 24, 2008 at 12:18:50PM +0300, Felipe Contreras wrote:
> > I developed a script that converts a monotone repository into a git
> > one (exact clone), I want to contribute it so everybody can use it.
> >
> > This is the gist of the script:
> >
> > mtn update --revision #{@id} --reallyquiet
> > git ls-files --modified --others --exclude-standard -z | git
> > update-index --add --remove -z --stdin
> > git write-tree
> > git write-raw < /tmp/commit.txt
> > git update-ref refs/mtn/#{@id} #{@git_id}
> >
> > branches.each do |e|
> >     git update-ref refs/heads/#{e} #{@git_id}
> > end
>
> You definitely want to use fast-import, but you probably want to do
> something a lot closer to fast-export for monotone (read: use its
> automate stdio interface and avoid expensive calls).
>
> Here's a simple monotone to git converter I wrote.  You'll need the
> Monotone::AutomateStdio perl module to use it (which I think I got it
> from monotone's net.venge.monotone.contrib.lib.automate-stdio branch).
> It is very fast; it can convert the OpenEmbedded repo in something like
> 5-10 minutes on my machine.
>
> Note that for monotone export to go fast you absolutely /must/ avoid the
> get_manifest operation.  In my converter I use the revision information
> directly.  Getting the renames right with this is a little tricky; IIRC,
> the ordering that works is:
>
> * Rename all renamed files, innermost files first, to temporary names.
> * Delete all deleted files, innermost first.
> * Rename all temporary names to permanent names, outermost first.
> * Add all new/modified files.
>
> Conveniently, all of the above can be done by directly emitting
> fast-import commands, so you don't have to keep track of trees directly.
> (With one exception, which I'll elaborate on in a different email.)
>
> -bcd

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: mtn-to-git.pl --]
[-- Type: text/x-perl; name=mtn-to-git.pl, Size: 5793 bytes --]

#!/usr/bin/perl
# Copyright (C) 2007-2008  Brian Downing
# This program is licensed under version 2 of the GNU GPL.

use strict;
use Monotone::AutomateStdio;
use Date::Parse;

my $D = 0;

my $m = Monotone::AutomateStdio->new($ARGV[0]);

my $revlist = [];
$m->graph($revlist);
my $sorted = [];

for my $rev (@$revlist) {
    push(@$sorted, $rev->{revision_id});
}

my $leaves = [];
$m->leaves($leaves);

$m->toposort($sorted, @$sorted);

my $marks = {};
my $mark = 1;
my $c = 0;
my %authors = {};

sub quote_file {
    $_ = shift;
    return $_;
    s/\\/\\\\/g;
    s/\n/\\n/g;
    s/"/\\"/g;
    return qq("$_");
}

sub lprint {
    my $fh = shift;
    print @_ if $D;
    print $fh @_;
}

sub lprintf {
    my $fh = shift;
    printf @_ if $D;
    printf $fh @_;
}

sub load_authors {
    my %authors = ();
    my $filename = shift(@_);
    open my $fi, '<', $filename or die "Could not open authors map file $filename\n"; 
    for my $line (<$fi>) {
        if ($line =~ m/(.*) <(.*)>/) {
            if ($2) {
                $authors{$2} = $1;
            }
        }
    }
    return %authors;
}

sub author_name {
    my $email = shift(@_);
    my $name = $authors{$email};
    if ($name) {
        return $name;
    } else {
        if ($email =~ m/(.+)\@.+/) {
            return $1;
        } else {
            return $email;
        }
    }
}

my $tmptag = "624d893e-ae1a-42d8-90a9-926a6ceffae8";

if ($ARGV[1]) {
    %authors = load_authors($ARGV[1]);
} else {
    %authors = {};
}
open my $fi, '|-', 'git-fast-import --export-marks=file';
# open my $fi, '>fast-import.dump';
binmode $fi;
for my $rev (@$sorted) {
    my ($time, $author, $msg) = ("0", "__UNKNOWN__", "__UNKNOWN__");
    my @certs;
    my @branches;
    $m->certs(\@certs, $rev);
    for my $cert (@certs) {
        my ($n, $v) = ($cert->{name}, $cert->{value});
        $author = $v if ($n eq 'author');
        $time = $v if ($n eq 'date');
        $msg = $v if ($n eq 'changelog');
        push(@branches, $v) if ($n eq 'branch');
    }
    my $email = $author;
    $msg .= "\nmtn-revision: $rev\n";
    for my $b (sort @branches) {
        $msg .= "mtn-branch: $b\n";
    }
    $time = str2time($time, 'UTC');
    my $mfest = [];
    $m->get_revision($mfest, $rev);
    my $orcount = 0;
    my $add_files = {};
    my $add_dirs = {};
    my $delete_files = {};
    my $from_tmpnames = {};
    my $to_tmpnames = {};
    my $curtmp = 0;
    my @parents;
    for my $e (@$mfest) {
        if ($e->{type} eq 'old_revision') {
            push(@parents, $e->{revision_id});
            ++$orcount;
        } 
        next if $orcount > 1;
        if ($e->{type} eq 'add_file' || $e->{type} eq 'patch') {
            my $id = $e->{file_id} || $e->{to_file_id};
            $add_files->{$e->{name}} = $id;
            unless ($marks->{$id}) {
                my $data;
                $m->get_file(\$data, $id);
                print "new file $id\n" if $D;
                print $fi "blob\n";
                my $len = length($data);
                print $fi "mark :$mark\n";
                $marks->{$id} = $mark++;
                print $fi "data $len\n$data\n";
                #print $fi "data $len\n";
                #print $fi pack('C', $data);
                #print $fi "\n";
            }
        } elsif ($e->{type} eq 'add_dir') {
            $add_dirs->{$e->{name}} = 1;
        } elsif ($e->{type} eq 'delete') {
            $delete_files->{$e->{name}} = 1;
        } elsif ($e->{type} eq 'rename') {
            $curtmp++;
            $from_tmpnames->{$e->{from_name}} = "__tmp_${tmptag}_$curtmp";
            $to_tmpnames->{$e->{to_name}} = "__tmp_${tmptag}_$curtmp";
        }
    }
    printf("rev $rev (%d/%d, %.2f%)\n",
           ++$c, scalar(@$sorted), 100*$c/scalar(@$sorted));
    print $fi "reset refs/import\n" unless @parents;
    lprint $fi, "commit refs/import\n";
    print $fi "mark :$mark\n";
    $marks->{$rev} = $mark++;
    if ($author =~ m(\s*(.*?\S)\s*<(.*)>\s*)) {
        # $author = $1;
        $email = $2;
    }
    # $author =~ s/[<>]/_/g;
    $email =~ s/[<>]/_/g;
    # $author =~ s/@.*//;
    $author = author_name($email);
    print $fi "committer $author <$email> $time +0000\n";
    my $len = length($msg);
    print $fi "data $len\n$msg\n";
    my $from = "from";
    for my $p (@parents) {
        # lprint $fi, "$from :$marks->{$p}\n";
        my $parent_mark = $marks->{$p};
        if ($parent_mark) {
            lprint $fi, "$from :$parent_mark\n";
            $from = "merge";
        }
    }
    for my $f (sort { length($b) <=> length ($a) } keys %$from_tmpnames) {
        lprintf($fi, "R %s %s\n",
                quote_file($f), quote_file($from_tmpnames->{$f}));
    }
    for my $f (sort { length($b) <=> length ($a) } keys %$delete_files) {
        lprintf($fi, "D %s\n", quote_file($f));
    }
    for my $f (sort { length($a) <=> length ($b) } keys %$to_tmpnames) {
        lprintf($fi, "R %s %s\n",
                quote_file($to_tmpnames->{$f}), quote_file($f));
    }
    for my $f (keys %$add_files) {
        lprintf($fi, "M 0644 :%s %s\n",
                $marks->{$add_files->{$f}}, quote_file($f));
    }
    for my $f (keys %$add_dirs) {
        $f .= "/" if $f;
        lprintf($fi, "M 0644 inline %s\n", quote_file("$f.gitignore"));
        lprint($fi, "data 0\n\n");
    }
    print $fi "\n";
}

my $branches = {};
for my $rev (@$leaves) {
    my $branch;
    my @certs;
    $m->certs(\@certs, $rev);
    for my $cert (@certs) {
        my ($n, $v) = ($cert->{name}, $cert->{value});
        $branch = $v if ($n eq 'branch');
    }
    my $r = $branches->{$branch};
    $branches->{$branch}--;
    if ($marks->{$rev}) {
        print $fi "reset refs/heads/$branch$r\n";
        print $fi "from :$marks->{$rev}\n\n";
    }
}

close $fi;