Incremental cvsimports

Incremental cvsimports

[Geoff Russell]

Dear Git,

I have a 3Gb repository (10 years of history) which I want to import into git.

I have tested little pieces of that and a 23Mb directory expands to 120Mb prior
to packing/pruning.  Therefore I don't have a big enough disk to import the
whole repository at once. So I figure I could do it in pieces with a pack after
each piece.

So I tried something like:

 1.      $ git cvsimport  -o austrics -v -d /cvsroot -C /GitRepo /cvsroot/A

followed by

 2.     $ git cvsimport  -o austrics -v -d /cvsroot -C /GitRepo /cvsroot/B

After 1, /GitRepo/.git is set up but /GitRepo has the members of A,
not A itself.
After 2, /GitRepo also has members of B. This isn't quite what I wanted!

Is there a way to do this?

Cheers,
Geoff Russell

Re: Incremental cvsimports

[Martin Langhoff]

On 5/24/06, Geoff Russell <geoffrey.russell@gmail.com> wrote:
> Dear Git,

Dear Geoff,

if you look at the list archive for the last couple of days, you'll
see there's been quite a bit of activity in tuning cvsimport so that
it scales better with large imports like yours. We have been playing
with a gentoo cvs repo with 300K commits / 1.6GB uncompressed.

Don't split up the tree... that'll lead to something rather ackward.
Instead, fetch and build git from Junio's 'master' branch which seems
to have collected most (all?) of the patches posted, including one
from Linus that will repack the repo every 1K commits -- keeping the
import size down.

You _will_ need a lot of memory though, as cvsps grows large (working
on a workaround now) and cvsimport grows a bit over time (where is
that last leak?!). And a fast machine -- specially fast IO. I've just
switched from an old test machine to an AMD64 with fast disks, and
it's importing around 10K commits per hour.

You will probably want to run cvsps by hand, and later use the -P flag.

cheers,


martin

Re: Incremental cvsimports

[Martin Langhoff]

On 5/24/06, Martin Langhoff <martin.langhoff@gmail.com> wrote:
> You _will_ need a lot of memory though, as cvsps grows large (working
> on a workaround now)

While the workaround is worked around, you can create the cvsps file
using something like

CVSROOT=~/tmp/gentoorepo/ cvsps --norc -x -u -A gentoo-x86  >
gentoo.cvsps 2> gentoo.err

and then pass a -P gentoo.cvsps to cvsimport. s/gentoo/myproject/ ;-)


martin

Re: Incremental cvsimports

[Geoff Russell]

[-- Attachment #1: Type: text/plain, Size: 1972 bytes --]

Dear Martin,


On 5/24/06, Martin Langhoff <martin.langhoff@gmail.com> wrote:
> On 5/24/06, Geoff Russell <geoffrey.russell@gmail.com> wrote:
> > Dear Git,
>
> Dear Geoff,
>
> if you look at the list archive for the last couple of days, you'll
> see there's been quite a bit of activity in tuning cvsimport so that
> it scales better with large imports like yours. We have been playing
> with a gentoo cvs repo with 300K commits / 1.6GB uncompressed.
>
> Don't split up the tree... that'll lead to something rather ackward.
> Instead, fetch and build git from Junio's 'master' branch which seems
> to have collected most (all?) of the patches posted, including one
> from Linus that will repack the repo every 1K commits -- keeping the
> import size down.

I got the latest git and yes, the size is kept down. I've only tried with
a smaller repository but it looks promising. When I ran git-cvsimport without a
CVS-module name (wanting the entire repository), it gave me a Usage message
indicating that the CVS-module name was optional - but it isn't :)

I did have to change
2 lines in git-cvsimport to get it to run with my 5.8.0 perl (problems with
POSIX errno). I've attached a patch but my work around isn't as quick as
what it replaced.

Many thanks, I'll have a go with the big repository at work tomorrow!

Cheers,
Geoff Russell

P.S. I've just started to look with git. We have wanted a cvs replacement for
a while but have been too scared to change (until now).



>
> You _will_ need a lot of memory though, as cvsps grows large (working
> on a workaround now) and cvsimport grows a bit over time (where is
> that last leak?!). And a fast machine -- specially fast IO. I've just
> switched from an old test machine to an AMD64 with fast disks, and
> it's importing around 10K commits per hour.

I

>
> You will probably want to run cvsps by hand, and later use the -P flag.
>
> cheers,
>
>
> martin
>
>

[-- Attachment #2: 999 --]
[-- Type: application/octet-stream, Size: 903 bytes --]

*** git-cvsimport	2006-05-24 20:13:19.000000000 +0930
--- /usr/local/bin/git-cvsimport	2006-05-24 20:22:27.000000000 +0930
*************** use File::Basename qw(basename dirname);
*** 23,29 ****
  use Time::Local;
  use IO::Socket;
  use IO::Pipe;
! use POSIX qw(strftime dup2 :errno_h);
  use IPC::Open2;
  
  $SIG{'PIPE'}="IGNORE";
--- 23,29 ----
  use Time::Local;
  use IO::Socket;
  use IO::Pipe;
! use POSIX qw(strftime dup2);
  use IPC::Open2;
  
  $SIG{'PIPE'}="IGNORE";
*************** sub get_headref ($$) {
*** 446,452 ****
  	    is_sha1($r) or die "Cannot get head id for $name ($r): $!";
  	    return $r;
      }
!     die "unable to open $f: $!" unless $! == POSIX::ENOENT;
      return undef;
  }
  
--- 446,452 ----
  	    is_sha1($r) or die "Cannot get head id for $name ($r): $!";
  	    return $r;
      }
!     die "unable to open $f: $!" if -f $f;
      return undef;
  }
  

Re: Incremental cvsimports

[Jeff King]

On Wed, May 24, 2006 at 08:49:03PM +0930, Geoff Russell wrote:

> I did have to change 2 lines in git-cvsimport to get it to run with my
> 5.8.0 perl (problems with POSIX errno). I've attached a patch but my
> work around isn't as quick as what it replaced.

Can you describe your problem in more detail? The POSIX errno constants
have been available since long before 5.8.0, so we should be able to use
them.

(btw, the change was introduced in my commit() cleanups:
  e73aefe4fdba0d161d9878642c69b40d83a0204c).

-Peff

Re: Incremental cvsimports

[Geoff Russell]

Dear Jeff,

See below.

On 5/24/06, Jeff King <peff@peff.net> wrote:
> On Wed, May 24, 2006 at 08:49:03PM +0930, Geoff Russell wrote:
>
> > I did have to change 2 lines in git-cvsimport to get it to run with my
> > 5.8.0 perl (problems with POSIX errno). I've attached a patch but my
> > work around isn't as quick as what it replaced.
>
> Can you describe your problem in more detail? The POSIX errno constants
> have been available since long before 5.8.0, so we should be able to use
> them.


   $ ./git-cvsimport

   ":errno_h" is not exported by the POSIX module
   Can't continue after import errors at
/usr/lib/perl5/5.8.0/i386-linux-thread-multi/POSIX.pm line 19
    BEGIN failed--compilation aborted at ./git-cvsimport line 26.

When I deleted ":errno_h" I needed to patch the place it was used (as per patch
I attached in original post).

Cheers,
Geoff Russell



>
> (btw, the change was introduced in my commit() cleanups:
>   e73aefe4fdba0d161d9878642c69b40d83a0204c).
>
> -Peff
> -
> To unsubscribe from this list: send the line "unsubscribe git" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>

Re: Incremental cvsimports

[Jeff King]

On Wed, May 24, 2006 at 10:03:44PM +0930, Geoff Russell wrote:

>   ":errno_h" is not exported by the POSIX module
>   Can't continue after import errors at
> /usr/lib/perl5/5.8.0/i386-linux-thread-multi/POSIX.pm line 19
>    BEGIN failed--compilation aborted at ./git-cvsimport line 26.

Hmm. It looks like something is nonstandard in your setup. I just compiled
5.8.0 from source and the :errno_h tag works fine. What is your
platform?  Can you try the following and let me know which work:
  $ perl -e 'use POSIX qw(:errno_h)'
  $ perl -e 'use POSIX qw(errno_h)'
  $ perl -e 'use Errno'

-Peff

Re: Incremental cvsimports

[Geoff Russell]

Hi Jeff,

On 5/24/06, Jeff King <peff@peff.net> wrote:
> On Wed, May 24, 2006 at 10:03:44PM +0930, Geoff Russell wrote:
>
> >   ":errno_h" is not exported by the POSIX module
> >   Can't continue after import errors at
> > /usr/lib/perl5/5.8.0/i386-linux-thread-multi/POSIX.pm line 19
> >    BEGIN failed--compilation aborted at ./git-cvsimport line 26.
>
> Hmm. It looks like something is nonstandard in your setup. I just compiled
> 5.8.0 from source and the :errno_h tag works fine. What is your
> platform?  Can you try the following and let me know which work:

I compiled perl from source on Mandrake 9.1.

>   $ perl -e 'use POSIX qw(:errno_h)'
>   $ perl -e 'use POSIX qw(errno_h)'
>   $ perl -e 'use Errno'

All 3 work.  But if I add a second tag before the ':errno_h", then I
get an error.

The "use" line that makes git-cvsimport compile for me is:

        use POSIX qw(strftime dup2 ENOENT);

Which just imports the required symbol and not the full tag list.

Cheers,
Geoff.

>
> -Peff
> -
> To unsubscribe from this list: send the line "unsubscribe git" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>

Re: Incremental cvsimports

[Jeff King]

On Wed, May 24, 2006 at 11:17:32PM +0930, Geoff Russell wrote:

> All 3 work.  But if I add a second tag before the ':errno_h", then I
> get an error.
> 
> The "use" line that makes git-cvsimport compile for me is:
> 
>        use POSIX qw(strftime dup2 ENOENT);

Odd. It's either a bug with importing tags in older versions, or there's
some deep perl voodoo that I don't understand (either way, it is "fixed"
in more recent versions).  Importing ENOENT directly is reasonable.

Junio, can you apply the following fix?

diff --git a/git-cvsimport.perl b/git-cvsimport.perl
index af331d9..76f6246 100755
--- a/git-cvsimport.perl
+++ b/git-cvsimport.perl
@@ -23,7 +23,7 @@ use File::Basename qw(basename dirname);
 use Time::Local;
 use IO::Socket;
 use IO::Pipe;
-use POSIX qw(strftime dup2 :errno_h);
+use POSIX qw(strftime dup2 ENOENT);
 use IPC::Open2;
 
 $SIG{'PIPE'}="IGNORE";

Re: Incremental cvsimports

[Junio C Hamano]

Jeff King <peff@peff.net> writes:

> Odd. It's either a bug with importing tags in older versions, or there's
> some deep perl voodoo that I don't understand (either way, it is "fixed"
> in more recent versions).  Importing ENOENT directly is reasonable.

Sounds good.  Thanks for the back-and-forth helping others in
the community.  I appreciate it.

> Junio, can you apply the following fix?

Will do, but I would have preferred if you did the commit log
message and the stuff properly.  Less work for me ;-).

>
> diff --git a/git-cvsimport.perl b/git-cvsimport.perl
> index af331d9..76f6246 100755
> --- a/git-cvsimport.perl
> +++ b/git-cvsimport.perl
> @@ -23,7 +23,7 @@ use File::Basename qw(basename dirname);
>  use Time::Local;
>  use IO::Socket;
>  use IO::Pipe;
> -use POSIX qw(strftime dup2 :errno_h);
> +use POSIX qw(strftime dup2 ENOENT);
>  use IPC::Open2;
>  
>  $SIG{'PIPE'}="IGNORE";