cvsimport: trying to convert freebsd cvs to git

cvsimport: trying to convert freebsd cvs to git

[Miklos Vajna]

[-- Attachment #1: Type: text/plain, Size: 1507 bytes --]

hi,

recently Stefan reported that he tried to convert the freebsd cvs' src
module to git and he failed. i tried to help him, but i failed, too.
here are my efforts:

he made the cvs available on a (relatively) fast rsync mirror at:

rsync://ftp.spline.de/FreeBSD-CVS

(so you don't have to use cvsup if it's a problem for you)

so after mirroring it, i had:

$ ls cvs
CVSROOT/  CVSROOT-ports/  CVSROOT-src/  ports/  src/

then i tried:

$ time git cvsimport -d `pwd`/cvs -C src.git src
Initialized empty Git repository in /home/vmiklos/git/freebsd/src.git/.git/
malformed revision
fatal: refs/heads/origin: not a valid SHA1
fatal: master: not a valid SHA1
warning: You appear to be on a branch yet to be born.
warning: Forcing checkout of HEAD.
fatal: just how do you expect me to merge 0 trees?
checkout failed: 256

real    15m11.529s
user    0m46.212s
sys     0m6.680s

my questions:

1) does cvsimport supports the case when the source if on the local
filesystem, and not in not on a cvs server?

first i wanted to make sure that cvsimport itself works properly here:

$ git cvsimport -d :pserver:anonymous@tcpflow.cvs.sourceforge.net:/cvsroot/tcpflow -C tcpflow tcpflow

and it converted this small repo fine

2) if it supports, then i think the real error message is 'malformed
revision'. what is the proper way to see where is that revision?

of course if cvsimport is not the right tool to incrementally convert
such a big repo, then i would be interested in other advices, too.

thanks,
- VMiklos

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: cvsimport: trying to convert freebsd cvs to git

[Anand Kumria]

On Sat, 22 Dec 2007 18:18:01 +0100, Miklos Vajna wrote:

> hi,
> 
> recently Stefan reported that he tried to convert the freebsd cvs' src
> module to git and he failed. i tried to help him, but i failed, too.
> here are my efforts:
> 
> he made the cvs available on a (relatively) fast rsync mirror at:
> 
> rsync://ftp.spline.de/FreeBSD-CVS
> 
> (so you don't have to use cvsup if it's a problem for you)
> 
> so after mirroring it, i had:
> 
> $ ls cvs
> CVSROOT/  CVSROOT-ports/  CVSROOT-src/  ports/  src/
> 

A few days prior to your email, someone else asked a similar question:

Some tools you can use:

git-cvsimport (via cvsps)
parsecvs
fromcvs
cvs2svn (development branch)

There is a more detailed listing on the wiki at
<http://git.or.cz/gitwiki/
InterfacesFrontendsAndTools#head-8870e1c81cc93f9a7a7acb5e969924ee60182d6b>

If you have access to the raw ",v" files as you do, I would suggest you 
try out parsecvs. It will not do incremental imports but will you a good 
idea of how you can expect the converted repository to behave / act.

Anand

Re: cvsimport: trying to convert freebsd cvs to git

[David Soria Parra]

> If you have access to the raw ",v" files as you do, I would suggest you 
> try out parsecvs. It will not do incremental imports but will you a good 
> idea of how you can expect the converted repository to behave / act.


ParseCVS works out fine with the PHP repository for us while cvsimport
has some problems with branches. For me it looks like doing parsecvs on
an existing repository tries to check all ,v files but in fact producing
a more or less incremental update.

If you try to use parsecvs make sure that your branchname doesnot
contain / or ~ as this will cause an error on parsecvs. If you need it,
we have a simple patch for that with a small workaround on that.

Re: cvsimport: trying to convert freebsd cvs to git

[Jeff King]

On Sat, Dec 22, 2007 at 06:18:01PM +0100, Miklos Vajna wrote:

> then i tried:
> 
> $ time git cvsimport -d `pwd`/cvs -C src.git src
> Initialized empty Git repository in /home/vmiklos/git/freebsd/src.git/.git/
> malformed revision

This is the actual error; everything else is git-cvsimport failing to
notice the problem and continuing anyway.

I will take a look and see if it is simple to stop the process here and
produce a better error message.

> 1) does cvsimport supports the case when the source if on the local
> filesystem, and not in not on a cvs server?

Yes, it should work fine (it will still call cvsps and CVS, so it should
look the same to git-cvsimport).

> 2) if it supports, then i think the real error message is 'malformed
> revision'. what is the proper way to see where is that revision?

Try git-cvsimport -v.

-Peff

Re: cvsimport: trying to convert freebsd cvs to git

[Martin Langhoff]

On Dec 24, 2007 6:29 AM, Jeff King <peff@peff.net> wrote:
> On Sat, Dec 22, 2007 at 06:18:01PM +0100, Miklos Vajna wrote:
>
> > then i tried:
> >
> > $ time git cvsimport -d `pwd`/cvs -C src.git src
> > Initialized empty Git repository in /home/vmiklos/git/freebsd/src.git/.git/
> > malformed revision
>
> This is the actual error; everything else is git-cvsimport failing to
> notice the problem and continuing anyway.
>
> I will take a look and see if it is simple to stop the process here and
> produce a better error message.

Indeed. Running verbose will name the branch name that triggers that
error. One thing I find useful is to capture the commandline passed to
cvsps (using ps), stopping cvsimport+cvsps, and running cvsps by hand
with the same commandline (remember to set TZ=UTC in the
environment!), saving STDOUT to a file.

Once you have the file, you can feed it to cvsimport with the -P
parameter (IIRC). Usually cvsimport errors are all about wonky cvsps
output.

For the record, I normally do my initial imports with Keith's
parsecvs, and then run incrementals using git-cvsimport. These
incrementals aren't 100% perfect, so I routinely check for 'drift'.

cheers,


m

Re: cvsimport: trying to convert freebsd cvs to git

[Jeff King]

On Sat, Dec 22, 2007 at 06:18:01PM +0100, Miklos Vajna wrote:

> $ time git cvsimport -d `pwd`/cvs -C src.git src
> Initialized empty Git repository in /home/vmiklos/git/freebsd/src.git/.git/
> malformed revision
> fatal: refs/heads/origin: not a valid SHA1
> fatal: master: not a valid SHA1
> warning: You appear to be on a branch yet to be born.
> warning: Forcing checkout of HEAD.
> fatal: just how do you expect me to merge 0 trees?
> checkout failed: 256

I looked into this a bit more. The error is definitely coming from
cvsps. I can't even get a reasonable cvsps output for this repo
(presumably because of behind-the-scenes hackery on the RCS files).
E.g., try making a module with _just_
"src/release/Attic/MIRROR.SITES,v"; I can't even get a reasonable cvsps
output on that.

You might have better results with one of the other importers that have
been suggested, most of which are more robust in the face of CVS
oddities.

That being said, it's awful that git-cvsimport doesn't stop when cvsps
exits with an error, producing all of those meaningless errors. Patch
will follow.

-Peff

[PATCH] cvsimport: die on cvsps errors

[Jeff King]

We were not previously checking the exit status of cvsps at
all. If it exited before producing any useful output, we
ended up with an empty import, which caused a spew of
confusing error messages from other parts of git:

$ git-cvsimport foo
Initialized empty Git repository in ...
some error from cvsps
fatal: refs/heads/origin: not a valid SHA1
fatal: master: not a valid SHA1
warning: You appear to be on a branch yet to be born.
warning: Forcing checkout of HEAD.
fatal: just how do you expect me to merge 0 trees?
checkout failed: 256

Now we get:

$ git-cvsimport foo
Initialized empty Git repository in ...
some error from cvsps
git-cvsimport: fatal: cvsps reported error

Signed-off-by: Jeff King <peff@peff.net>
---
On Sun, Dec 23, 2007 at 10:05:51PM -0500, Jeff King wrote:

> That being said, it's awful that git-cvsimport doesn't stop when cvsps
> exits with an error, producing all of those meaningless errors. Patch
> will follow.

And here it is.

 git-cvsimport.perl |    1 +
 1 files changed, 1 insertions(+), 0 deletions(-)

diff --git a/git-cvsimport.perl b/git-cvsimport.perl
index 1fa9a22..6d8ff93 100755
--- a/git-cvsimport.perl
+++ b/git-cvsimport.perl
@@ -632,6 +632,7 @@ unless ($opt_P) {
 	    print $cvspsfh $_;
 	}
 	close CVSPS;
+	$? == 0 or die "git-cvsimport: fatal: cvsps reported error\n";
 	close $cvspsfh;
 } else {
 	$cvspsfile = $opt_P;
-- 
1.5.4.rc1.1119.g1e6bc-dirty

Re: [PATCH] cvsimport: die on cvsps errors

[Martin Langhoff]

> We were not previously checking the exit status of cvsps
> at all.

Acked. We should have done this ages ago. Thanks!

m

Re: [PATCH] cvsimport: die on cvsps errors

[Miklos Vajna]

[-- Attachment #1: Type: text/plain, Size: 301 bytes --]

On Sun, Dec 23, 2007 at 10:08:19PM -0500, Jeff King <peff@peff.net> wrote:
> > That being said, it's awful that git-cvsimport doesn't stop when cvsps
> > exits with an error, producing all of those meaningless errors. Patch
> > will follow.
> 
> And here it is.

great, thanks :)

- VMiklos

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]