Towards CVS code-exchange and gateways

Towards CVS code-exchange and gateways

[Martin Langhoff]

Now that I have a few cvs2git gateways (git-cvsimport running on cron)
for the projects we work on, I am starting to need tools to "replay"
chunks of git history against external CVS repositories. So I am
thinking of writing two new scripts which expect to be executed inside
a CVS working copy:

 + "git-cvsapplypatch < patchfile" takes a patch as formatted by
git-format-patch, apply using patch (with options to be strict of
fuzzy), prepares the commit message and (optionally) autocommit if
patch returned clean.

 + "GIT_DIR=~/foo/bar/.git git-cvsreplaycommit parent:child" takes a
pair of git revisions that must be parent/child, applies the diff on
text files and changes on binary objects. Prepares the commit message
and (optionally) autocommits if the merge was clean.

Which should lead later to a git-cvsapplymbox script to automate the
process further. This part of the automation is actually a bit scary:
with git, all your scripted merges take place in your private repo,
and you review the result of the whole patchrun before pushing it out
to a public repo. With cvs, if the merge turns out to apply cleanly
but be a really bad idea... it'll be way too late. I hope to be able
to find a smart way to run it in "test" mode.

Is there anyone working on a git -> cvs gateway or similar scripts?

cheers,


martin

Re: Towards CVS code-exchange and gateways

[Junio C Hamano]

Martin Langhoff <martin.langhoff@gmail.com> writes:

> Is there anyone working on a git -> cvs gateway or similar scripts?

Working on, no, but I was thinking about something like that at
work (today is my non-git day but I use git on top of CVS to
track my private changes in my CVS work tree which is my git
repo there).

I maintain at 'cvs' and 'master' branch (and other git branches)
there, and always 'git-checkout cvs' before doing 'cvs update',
and do 'git-commit' to slurp in the changes to the cvs side.
Then 'git checkout master ; git pull . cvs' to sync the master
side, work on a bit, commit.  Showing what I've done to the CVS
side, currently I do something like this:

      $ git-checkout cvs
      $ git pull . master
      $ git log ORIG_HEAD.. >L
      $ cvs commit -F L

This isn't that bad because I do not care much about how the
development history on the CVS side looks like.  But propagating
each git commit separately to CVS side would be much better, and
what you outlined should work nicely.

Re: Towards CVS code-exchange and gateways

[Martin Langhoff]

Update here: Sven sent me a cool 4 line shell script that has the
basics. Next thing I know, and I was dumped into a boring presentation
-- so I turned it into a more complete Perl script. Still incomplete &
extremely untested, but showing the basics.

Give me a couple more boring presentations and we'll be done ;-)

cheers,


martin
---
#!/usr/bin/perl -w

use strict;
use Getopt::Std;

unless ($ENV{GIT_DIR} && -r $ENV{GIT_DIR}){
    die "GIT_DIR is not defined or is unreadable";
}

our ($opt_h, $opt_p);

getopt('hp');

$opt_h && usage();

die "Need at least one commit identifier!" unless @ARGV;

# resolve target commit
my $commit;
$commit = pop @ARGV;
$commit = `git-rev-parse --verify "$commit"^0"`;
chomp $commit;
if ($?) {
    die "The commit reference did not resolve!";
}

# resolve what parent we want
my $parent;
if (@ARGV) {
    $parent = pop @ARGV;
    $parent =  `git-rev-parse --verify "$parent"^0"`;
    chomp $parent;
    if ($?) {
	die "The parent reference did not resolve!";
    }
}

# parents from the commit itself
my @parents = `git-cat-file commmit $commit | grep -E '^parent
\\w{40}\$' | sed -e 's/^parent //'`;
@parents = map { chomp } @parents;

if ($parent) {
    # double check that it's a valid parent
    foreach my $p (@parents) {
	my $found;
	if ($p eq $parent) {
	    $found = 1;
	    last;
	}; # found it
	die "Did not find $parent in the parents for this commit!";
    }
} else { # we don't have a parent from the cmdline...
    if (@parents == 1) { # it's safe to get it from the commit
	$parent = $parents[0];
    } else { # or perhaps not!
	die "This commit has more than one parent -- please name the parent
you want to use explicitly";
    }
}

# grab the commit message
`git-cat-file commit $commit | sed -e '1,/^$/d' > .msg`;
$? && die "Error extraction the commit message";

my @files = `git-diff-tree -r $parent $commit | cut -f 2`;
$? && die "Error in git-diff-tree";
@files = map { chomp } @files;

# check that the files are clean and up to date according to cvs
my $dirty;
foreach my $f (@files) {
    # TODO:we need to handle removed in cvs and/or new (from git)
    my $status = `cvs status $f`;

    unless ($status =~ m/Up to date/) {
	$dirty = 1;
	warn "File $f not up to date!\n";
    }
}
if ($dirty) {
    die "Exiting: your CVS tree is not clean for this merge.";
}

###
### NOTE: if you are planning to die() past this point
###       you MUST call cleanupcvs(@files) before die()
###

## apply changes to binary files
my @bfiles = `git-diff-tree -p $parent $commit | grep '^Binary'`;
@bfiles = map { chomp } @bfiles;
foreach my $f (@bfiles) {
    # check that the file in cvs matches the "old" file

    # replace with the new file

}

## apply non-binary changes
## (git-diff-tree -p $1 | patch -p1) && cvs commit -F .msg


sub usage {
	print STDERR <<END;
Usage: GIT_DIR=/path/to/.gi ${\basename $0}      # fetch/update GIT from CVS
       [-h] [-p] [ parent ] commit
END
	exit(1);
}

# ensure cvs is clean before we die
sub cleanupcvs {
    my @files = @_;
    foreach my $f (@files) {
	`cvs -q update -C "$f"`;
	if ($?) {
	    warn "Warning! Failed to cleanup state of $f\n";
	}
    }
}

Re: Towards CVS code-exchange and gateways

[Petr Baudis]

Dear diary, on Wed, Oct 26, 2005 at 05:35:41AM CEST, I got a letter
where Martin Langhoff <martin.langhoff@gmail.com> told me that...
> Update here: Sven sent me a cool 4 line shell script that has the
> basics. Next thing I know, and I was dumped into a boring presentation
> -- so I turned it into a more complete Perl script. Still incomplete &
> extremely untested, but showing the basics.
> 
> Give me a couple more boring presentations and we'll be done ;-)

Can I then import from the CVS incrementally later? I guess from the
cvsimport code that it just tries to always import everything but the
already imported commits get the same hashes so it magically imports
"incrementally"... (The code could use some descriptive comments, it is
rather spaggetish.)

If someone really desperately needs this, BTW, you might be able to
merge two Monotone branches (.git and .cvssync) to get two-way
incremental GIT and CVS interface, and then do that through Monotone.
;-))

-- 
				Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
VI has two modes: the one in which it beeps and the one in which
it doesn't.

Re: Towards CVS code-exchange and gateways

[Martin Langhoff]

[-- Attachment #1: Type: text/plain, Size: 1976 bytes --]

On 10/26/05, Petr Baudis <pasky@suse.cz> wrote:
> Dear diary, on Wed, Oct 26, 2005 at 05:35:41AM CEST, I got a letter
> where Martin Langhoff <martin.langhoff@gmail.com> told me that...
> > Update here: Sven sent me a cool 4 line shell script that has the
> > basics. Next thing I know, and I was dumped into a boring presentation
> > -- so I turned it into a more complete Perl script. Still incomplete &
> > extremely untested, but showing the basics.
> >
> > Give me a couple more boring presentations and we'll be done ;-)
>
> Can I then import from the CVS incrementally later? I guess from the
> cvsimport code that it just tries to always import everything but the
> already imported commits get the same hashes so it magically imports
> "incrementally"... (The code could use some descriptive comments, it is
> rather spaggetish.)

Yes -- the cvsimport script does incremental imports. If you use
cvsimport to bring in the commits from CVS, you must treat those
branches as "read only". But you can open new heads from them and do
your "local" work there.

That's what we do with Moodle, as you can see here:
http://locke.catalyst.net.nz/gitweb?p=moodle.git;a=summary the
branches in uppercase are imported from CVS by a cronjob. The branches
that begin with "mdl" open off those, and we merge the cvs updates
often.

The goal for this script that I'm drafting is to be able to push
commits back into cvs in a format that maximises the chance of
git-cherry identifying them when they are echoed back (and thus
avoiding bogus conflicts).

> If someone really desperately needs this, BTW, you might be able to
> merge two Monotone branches (.git and .cvssync) to get two-way
> incremental GIT and CVS interface, and then do that through Monotone.
> ;-))

I'm really scared by the concept ;-)

The script has moved forward quite a bit -- and I suspect it might
even work. Current (untested) version attached.

cheers,

martin

[-- Attachment #2: git-cvsapplycommmit.perl --]
[-- Type: application/octet-stream, Size: 4013 bytes --]

#!/usr/bin/perl -w

use strict;
use Getopt::Std;
use File::Temp qw(tempdir);

unless ($ENV{GIT_DIR} && -r $ENV{GIT_DIR}){
    die "GIT_DIR is not defined or is unreadable";
}

our ($opt_h, $opt_p);

getopt('hp');

$opt_h && usage();

die "Need at least one commit identifier!" unless @ARGV;

# setup a tempdir
our ($tmpdir, $tmpdirname) = tempdir('git-cvsapplycommit-XXXXXX',
				     TMPDIR => 1,
				     CLEANUP => 1);

# resolve target commit
my $commit;
$commit = pop @ARGV;
$commit = `git-rev-parse --verify "$commit"^0"`;
chomp $commit;
if ($?) {
    die "The commit reference did not resolve!";
}

# resolve what parent we want
my $parent;
if (@ARGV) {
    $parent = pop @ARGV;
    $parent =  `git-rev-parse --verify "$parent"^0"`;
    chomp $parent;
    if ($?) {
	die "The parent reference did not resolve!";
    }
}

# find parents from the commit itself
my @parents = `git-cat-file commmit $commit | grep -E '^parent \\w{40}\$' | sed -e 's/^parent //'`;
@parents = map { chomp } @parents;

if ($parent) {
    # double check that it's a valid parent
    foreach my $p (@parents) {
	my $found;
	if ($p eq $parent) {
	    $found = 1;
	    last;
	}; # found it
	die "Did not find $parent in the parents for this commit!";
    }
} else { # we don't have a parent from the cmdline...
    if (@parents == 1) { # it's safe to get it from the commit
	$parent = $parents[0];
    } else { # or perhaps not!
	die "This commit has more than one parent -- please name the parent you want to use explicitly";
    }
}

# grab the commit message
`git-cat-file commit $commit | sed -e '1,/^$/d' > .msg`;
$? && die "Error extraction the commit message";

my @files = `git-diff-tree -r $parent $commit | cut -f 2`;
$? && die "Error in git-diff-tree";
@files = map { chomp } @files;

# check that the files are clean and up to date according to cvs
my $dirty;
foreach my $f (@files) {
    # TODO:we need to handle removed in cvs and/or new (from git) 
    my $status = `cvs -q status "$f" | grep '^File: '`;

    unless ($status =~ m/Status: Up-to-date$/) {
	$dirty = 1;
	warn "File $f not up to date in your CVS checkout!\n";
    }
}
if ($dirty) {
    die "Exiting: your CVS tree is not clean for this merge.";
}

###
### NOTE: if you are planning to die() past this point
###       you MUST call cleanupcvs(@files) before die()
###


print "'Patching' binary files\n";

my @bfiles = `git-diff-tree -p $parent $commit | grep '^Binary'`;
@bfiles = map { chomp } @bfiles;
foreach my $f (@bfiles) {
    # check that the file in cvs matches the "old" file
    # extract the file to $tmpdir and comparre with cmp
    my $tree = `git-rev-parse $parent^{tree} `;
    chomp $tree;
    my $blob = `git-ls-tree $tree "$f" | cut -f 1 | cut -d ' ' -f 3`;
    chomp $blob;
    `git-cat-file blob $blob > $tmp/blob`;
    `cmp -q $f $tmp/blob`;
    if ($?) {
	warn "Binary file $f in CVS does not match parent.\n";
	$dirty = 1;
	next;
    }

    # replace with the new file
     `git-cat-file blob $blob > $f`;

    # TODO: something smart with file modes

}
if ($dirty) {
    cleanupcvs(@files);
    die "Exiting: Binary files in CVS do not match parent";
}

## apply non-binary changes
my $fuzz = $opt_p ? 0 : 2;

print "Patching non-binary files\n";
print `(git-diff-tree -p $1 | patch -p1 -F $fuzz ) 2>&1`;
if ($?) {
    cleanupcvs(@files);
    die "Exiting: Patch did not succeed -- you will have to apply this patch manually";
}

print "Commit to CVS\n";
my $commitfiles = join(' ', @files);
print `cvs commit -F .msg $commitfiles 2>&1`;
if ($?) {
    cleanupcvs(@files);
    die "Exiting: The commit did not succeed";
}
print "Committed successfully to CVS\n";

sub usage {
	print STDERR <<END;
Usage: GIT_DIR=/path/to/.gi ${\basename $0}      # fetch/update GIT from CVS
       [-h] [-p] [ parent ] commit
END
	exit(1);
}

# ensure cvs is clean before we die
sub cleanupcvs {
    my @files = @_;
    foreach my $f (@files) {
	`cvs -q update -C "$f"`;
	if ($?) {
	    warn "Warning! Failed to cleanup state of $f\n";
	}
    }
}

Re: Towards CVS code-exchange and gateways

[Petr Baudis]

Dear diary, on Wed, Oct 26, 2005 at 11:11:49AM CEST, I got a letter
where Martin Langhoff <martin.langhoff@gmail.com> told me that...
> The goal for this script that I'm drafting is to be able to push
> commits back into cvs in a format that maximises the chance of
> git-cherry identifying them when they are echoed back (and thus
> avoiding bogus conflicts).

Aha, so you are not aiming for proper two-way incremental i/e, and one
will have to cherrypick to import after an export... well, I guess that
can be good enough for many cases. But to use Linus' words, the really
interesting problem is to have the proper revision tree in the CVS heads
as well, so that you could do normal merges. And it shouldn't be _that_
hard either...

> > If someone really desperately needs this, BTW, you might be able to
> > merge two Monotone branches (.git and .cvssync) to get two-way
> > incremental GIT and CVS interface, and then do that through Monotone.
> > ;-))
> 
> I'm really scared by the concept ;-)

I *think* someone actually really did something like that. ;-)

-- 
				Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
VI has two modes: the one in which it beeps and the one in which
it doesn't.