Importing Mozilla CVS into git

Importing Mozilla CVS into git

[Jon Smirl]

I've been working on importing the Mozilla CVS into git for the last
few days. I've fixed up parsecvs so that it can parse the entire
repository without errors. Now I'm running into problems because there
are over 300 branches.

I just run into a problem with git show-branch. Mozilla CVS has a lot
more than 29 refs, is this something that can be expanded?

Is anyone interested in helping out with this? My knowledge of git and
CVS is limited. Mozilla CVS is about 3GB and it is available via
rsync. I can post the parsecvs changes if wanted.

rsync -az cvs-mirror.mozilla.org::mozilla ~/mozilla/cvs-mirror

-- 
Jon Smirl
jonsmirl@gmail.com

Re: Importing Mozilla CVS into git

[Keith Packard]

[-- Attachment #1: Type: text/plain, Size: 424 bytes --]

On Thu, 2006-06-01 at 18:21 -0400, Jon Smirl wrote:

> Is anyone interested in helping out with this? My knowledge of git and
> CVS is limited. Mozilla CVS is about 3GB and it is available via
> rsync. I can post the parsecvs changes if wanted.

Yes, please post parsecvs changes; I've been able to import several
repositories of this size without problems (given sufficient memory).

-- 
keith.packard@intel.com

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: Importing Mozilla CVS into git

[Jon Smirl]

[-- Attachment #1: Type: text/plain, Size: 1387 bytes --]

With the attached patch you can parse the entire Mozilla tree. The
tree has over 100,000 files in it and about 300 branches.

I ran this overnight and it failed with out of memory on a 1GB machine
with 1GB swap. If failed in the branch processing, the parse phase
succeeded.

How much memory does something like this need?

If you want to quickly check out the branch processing problems use
rsync to pull down just a copy of repository files.

I am getting 1000s of warnings like these and I haven't figured out why yet.

Warning: ../mozilla/mozilla/build/mac/build_scripts/Attic/MozillaCheckoutList.txt,v:
unnamed branch from master
Warning: ../mozilla/mozilla/build/unix/run-mozilla.sh,v: unnamed
branch from master
Warning: ../mozilla/mozilla/Makefile.in,v: unnamed branch from master of 99855
Warning: ../mozilla/mozilla/Makefile.in,v: unnamed branch from master
Warning: ../mozilla/mozilla/allmakefiles.sh,v: unnamed branch from master99855
Warning: ../mozilla/mozilla/allmakefiles.sh,v: unnamed branch from master
Warning: ../mozilla/mozilla/cmd/macfe/MailNews/AddressBook/Attic/UAddressBookUtilities.cp,v:
unnamed branch from master
Warning: ../mozilla/mozilla/cmd/macfe/MailNews/AddressBook/Attic/UAddressBookUtilities.h,v:
unnamed branch from master
Warning: ../mozilla/mozilla/cmd/macfe/central/Attic/msv2dsk.cp,v:
unnamed branch from master

-- 
Jon Smirl
jonsmirl@gmail.com

[-- Attachment #2: patch --]
[-- Type: application/octet-stream, Size: 1618 bytes --]

diff --git a/gram.y b/gram.y
index 502991e..4566628 100644
--- a/gram.y
+++ b/gram.y
@@ -18,6 +18,7 @@
  */
 
 #include "cvs.h"
+#define YYDEBUG 1
     
 void yyerror (char *msg);
 
@@ -72,6 +73,7 @@ header		: HEAD NUMBER SEMI
 		| symbollist
 		  { this_file->symbols = $1; }
 		| LOCKS SEMI lock_type SEMI
+		| LOCKS NAME COLON NUMBER SEMI lock_type SEMI
 		| COMMENT DATA SEMI
 		| EXPAND DATA SEMI
 		;
@@ -91,6 +93,21 @@ symbol		: NAME COLON NUMBER
 			$$->name = $1;
 			$$->number = $3;
 		  }
+		| NUMBER COLON NUMBER
+		  {
+		  	char buffer[200];
+		  	int i, j;
+		  	
+		  	j = 0;
+		  	for (i = 0; i < $1.c; i++) {
+		  		if (i != 0)
+			  		j += sprintf(&buffer[j], ".");
+		  		j += sprintf(&buffer[j], "%d", $1.n[i]);
+		  	}
+			$$ = calloc (1, sizeof (cvs_symbol));
+			$$->name = atom(buffer);
+			$$->number = $3;
+		  }
 		;
 revisions	: revision revisions
 		  { $1->next = $2; $$ = $1; }
diff --git a/lex.l b/lex.l
index f25b8a5..956eed7 100644
--- a/lex.l
+++ b/lex.l
@@ -52,7 +52,7 @@ #define YY_INPUT(buf,result,max_size) { 
 					BEGIN(INITIAL);
 					return TEXT_DATA;
 				}
-<CONTENT>[-a-zA-Z_0-9+][-a-zA-Z_0-9+/]* {
+<CONTENT>[-a-zA-Z_+][-a-zA-Z_0-9+/.%]* {
 					yylval.s = atom (yytext);
 					return NAME;
 				}
diff --git a/parsecvs.c b/parsecvs.c
index df083c6..e9b8056 100644
--- a/parsecvs.c
+++ b/parsecvs.c
@@ -27,7 +27,9 @@ #endif
 
 cvs_file	*this_file;
 
-rev_execution_mode rev_mode = ExecuteGit;
+//rev_execution_mode rev_mode = ExecuteGit;
+//rev_execution_mode rev_mode = ExecuteGraph;
+rev_execution_mode rev_mode = ExecuteSplits;
 
 int elide = 0;
 int difffiles = 1;

Re: Importing Mozilla CVS into git

[Keith Packard]

[-- Attachment #1: Type: text/plain, Size: 2212 bytes --]

On Thu, 2006-06-01 at 20:55 -0400, Jon Smirl wrote:
> With the attached patch you can parse the entire Mozilla tree. The
> tree has over 100,000 files in it and about 300 branches.

that's good news.

> I ran this overnight and it failed with out of memory on a 1GB machine
> with 1GB swap. If failed in the branch processing, the parse phase
> succeeded.

yeah, parsecvs has some internal storage inefficiencies which need to be
addressed. In particular, every commit has a pointer to the related
revision of every file in the commit. Much like git used to store every
filename in the commit object and was changed to share common directory
contents, parsecvs should be fixed to do the same.

> How much memory does something like this need?

It's basically 4 * nrev * nfile bytes on a 32-bit machine, multiply by 2
for a 64-bit machine.

> If you want to quickly check out the branch processing problems use
> rsync to pull down just a copy of repository files.
> 
> I am getting 1000s of warnings like these and I haven't figured out why yet.
> 
> Warning: ../mozilla/mozilla/build/mac/build_scripts/Attic/MozillaCheckoutList.txt,v:
> unnamed branch from master
> Warning: ../mozilla/mozilla/build/unix/run-mozilla.sh,v: unnamed
> branch from master
> Warning: ../mozilla/mozilla/Makefile.in,v: unnamed branch from master of 99855
> Warning: ../mozilla/mozilla/Makefile.in,v: unnamed branch from master
> Warning: ../mozilla/mozilla/allmakefiles.sh,v: unnamed branch from master99855
> Warning: ../mozilla/mozilla/allmakefiles.sh,v: unnamed branch from master
> Warning: ../mozilla/mozilla/cmd/macfe/MailNews/AddressBook/Attic/UAddressBookUtilities.cp,v:
> unnamed branch from master
> Warning: ../mozilla/mozilla/cmd/macfe/MailNews/AddressBook/Attic/UAddressBookUtilities.h,v:
> unnamed branch from master
> Warning: ../mozilla/mozilla/cmd/macfe/central/Attic/msv2dsk.cp,v:
> unnamed branch from master

yeah, these happen when vendor branches go awry. I'll pull the
repository and take a look. X.org had similar 'issues' as the current
CVS repo was built by merging mesa, XFree86 and X.org together in a
rather haphazard fashion.

-- 
keith.packard@intel.com

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: Importing Mozilla CVS into git

[Jon Smirl]

On 6/1/06, Keith Packard <keithp@keithp.com> wrote:
> yeah, these happen when vendor branches go awry. I'll pull the
> repository and take a look. X.org had similar 'issues' as the current
> CVS repo was built by merging mesa, XFree86 and X.org together in a
> rather haphazard fashion.

Let me know what you find. Converting this without rewriting parsecvs
looks to be beyond the capacity of my home machine. I'm sure you have
access to giant machines at Intel.

Did I see that you can use CVS client tools to manipulate a git
repository? Mozilla has a lot of users on other OSes besides Linux. It
would be nice to change the core server over to git and leave these
other users running their existing tools.

-- 
Jon Smirl
jonsmirl@gmail.com

Re: Importing Mozilla CVS into git

[Shawn Pearce]

Jon Smirl <jonsmirl@gmail.com> wrote:
> Did I see that you can use CVS client tools to manipulate a git
> repository? Mozilla has a lot of users on other OSes besides Linux. It
> would be nice to change the core server over to git and leave these
> other users running their existing tools.

Yes.  Look at git-cvsserver (ships standard as part of GIT).
It should also be faster than the original CVS server.  :-)

-- 
Shawn.

Re: Importing Mozilla CVS into git

[Keith Packard]

[-- Attachment #1: Type: text/plain, Size: 736 bytes --]

On Thu, 2006-06-01 at 22:36 -0400, Jon Smirl wrote:

> Did I see that you can use CVS client tools to manipulate a git
> repository? Mozilla has a lot of users on other OSes besides Linux. It
> would be nice to change the core server over to git and leave these
> other users running their existing tools.

It's possible, but I would not encourage people to use this for anything
other than passive monitoring of the code; CVS semantics are really too
weak to express the capabilities of the git repository, so changes made
through CVS will lose information.

Git runs fine on Windows these days; asking people to use reasonable
tools to contribute to the project doesn't seem crazy to me.

-- 
keith.packard@intel.com

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: Importing Mozilla CVS into git

[Jon Smirl]

On 6/1/06, Keith Packard <keithp@keithp.com> wrote:
> Git runs fine on Windows these days; asking people to use reasonable
> tools to contribute to the project doesn't seem crazy to me.

WIndows, Mac, Solaris and Linux will cover most Firefox developers.
Is git to go on those platforms? Is WIndows native or cygwin?

There are a few more people on weird platforms that will need a solution.
Are perl and shell script still a requirement?

-- 
Jon Smirl
jonsmirl@gmail.com

Re: Importing Mozilla CVS into git

[Keith Packard]

[-- Attachment #1: Type: text/plain, Size: 887 bytes --]

On Thu, 2006-06-01 at 23:47 -0400, Jon Smirl wrote:
> On 6/1/06, Keith Packard <keithp@keithp.com> wrote:
> > Git runs fine on Windows these days; asking people to use reasonable
> > tools to contribute to the project doesn't seem crazy to me.
> 
> WIndows, Mac, Solaris and Linux will cover most Firefox developers.
> Is git to go on those platforms? Is WIndows native or cygwin?

I think the windows stuff may still be cygwin, but Mac and Solaris work
fine with the git, of course. It's just simple posix code, after all

> There are a few more people on weird platforms that will need a solution.
> Are perl and shell script still a requirement?

Yeah, quite a bit of both of those still. Less over time as people
figure out that C code is generally easier to fix than a nasty
combination of C code, shell scripts and perl line noise.

-- 
keith.packard@intel.com

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: Importing Mozilla CVS into git

[Jon Smirl]

On 6/1/06, Keith Packard <keithp@keithp.com> wrote:
> On Thu, 2006-06-01 at 23:47 -0400, Jon Smirl wrote:
> > On 6/1/06, Keith Packard <keithp@keithp.com> wrote:
> > > Git runs fine on Windows these days; asking people to use reasonable
> > > tools to contribute to the project doesn't seem crazy to me.
> >
> > WIndows, Mac, Solaris and Linux will cover most Firefox developers.
> > Is git to go on those platforms? Is WIndows native or cygwin?
>
> I think the windows stuff may still be cygwin, but Mac and Solaris work
> fine with the git, of course. It's just simple posix code, after all

It is going to have to be native Windows to move some of the
developers over. They are true blue MS types that won't touch anything
close to Unix so cygwin is out.

-- 
Jon Smirl
jonsmirl@gmail.com

Re: Importing Mozilla CVS into git

[Shawn Pearce]

Jon Smirl <jonsmirl@gmail.com> wrote:
> On 6/1/06, Keith Packard <keithp@keithp.com> wrote:
> >On Thu, 2006-06-01 at 23:47 -0400, Jon Smirl wrote:
> >> On 6/1/06, Keith Packard <keithp@keithp.com> wrote:
> >> > Git runs fine on Windows these days; asking people to use reasonable
> >> > tools to contribute to the project doesn't seem crazy to me.
> >>
> >> WIndows, Mac, Solaris and Linux will cover most Firefox developers.
> >> Is git to go on those platforms? Is WIndows native or cygwin?
> >
> >I think the windows stuff may still be cygwin, but Mac and Solaris work
> >fine with the git, of course. It's just simple posix code, after all
> 
> It is going to have to be native Windows to move some of the
> developers over. They are true blue MS types that won't touch anything
> close to Unix so cygwin is out.

Then GIT on Windows might be out.

GIT today requires not only a decent UNIX shell but also, GNU tools,
Perl and Python.  Porting to Solaris has recently had some more
effort put into it to remove some of the GNU tool dependencies but
perhaps one of the most important features (git-merge-recursive)
is a Python script.

I'm running GIT at work on a Windows/Cygwin installation which is
really quite bare bones.  I think I have about 15 Cygwin packages
installed total and GIT is running fine in that environment.
It can't send patches by email but the corporate firewalls wouldn't
permit that anyway...

Perhaps you can tell the true blue MS types that Cygwin is a native
Windows application.  After all it uses the Win32 API.  :-)

-- 
Shawn.

Re: Importing Mozilla CVS into git

[Pavel Roskin]

On Fri, 2006-06-02 at 00:11 -0400, Shawn Pearce wrote:

> GIT today requires not only a decent UNIX shell but also, GNU tools,
> Perl and Python.  Porting to Solaris has recently had some more
> effort put into it to remove some of the GNU tool dependencies but
> perhaps one of the most important features (git-merge-recursive)
> is a Python script.

The great thing about git is that it's modular.  A single utility can be
replaced and retested in the same environment, without having to rewrite
the rest of the scripts.  A dedicated programmer with good C and Python
skills could rewrite git-merge-recursive.py in C in 2 days, I believe.
Add a few days of bug fixing, of course.

Dependency on Cygwin, Perl and Python is too much.  Windows is becoming
a legacy system in some circles, and it may run on legacy hardware.  Yet
it's irreplaceable as a testing platform for many projects.

I really need to rewrite git-clean in C, since it doesn't handle
embedded newlines properly.

-- 
Regards,
Pavel Roskin

Re: Importing Mozilla CVS into git

[Shawn Pearce]

Pavel Roskin <proski@gnu.org> wrote:
> On Fri, 2006-06-02 at 00:11 -0400, Shawn Pearce wrote:
> 
> > GIT today requires not only a decent UNIX shell but also, GNU tools,
> > Perl and Python.  Porting to Solaris has recently had some more
> > effort put into it to remove some of the GNU tool dependencies but
> > perhaps one of the most important features (git-merge-recursive)
> > is a Python script.
> 
> The great thing about git is that it's modular.  A single utility can be
> replaced and retested in the same environment, without having to rewrite
> the rest of the scripts.  A dedicated programmer with good C and Python
> skills could rewrite git-merge-recursive.py in C in 2 days, I believe.
> Add a few days of bug fixing, of course.

Heh.  Funny you should mention that.  I was just thinking a few
minutes ago about working on that exact change...
 
> Dependency on Cygwin, Perl and Python is too much.  Windows is becoming
> a legacy system in some circles, and it may run on legacy hardware.  Yet
> it's irreplaceable as a testing platform for many projects.

Its already legacy to me.  Heck its 2006 and my work desktop still
says something about 2000 when I login.  :-)
 
-- 
Shawn.

Re: Importing Mozilla CVS into git

[Johannes Schindelin]

Hi,

On Fri, 2 Jun 2006, Shawn Pearce wrote:

> Pavel Roskin <proski@gnu.org> wrote:
> > On Fri, 2006-06-02 at 00:11 -0400, Shawn Pearce wrote:
> > 
> > > GIT today requires not only a decent UNIX shell but also, GNU tools,
> > > Perl and Python.  Porting to Solaris has recently had some more
> > > effort put into it to remove some of the GNU tool dependencies but
> > > perhaps one of the most important features (git-merge-recursive)
> > > is a Python script.
> > 
> > The great thing about git is that it's modular.  A single utility can be
> > replaced and retested in the same environment, without having to rewrite
> > the rest of the scripts.  A dedicated programmer with good C and Python
> > skills could rewrite git-merge-recursive.py in C in 2 days, I believe.
> > Add a few days of bug fixing, of course.
> 
> Heh.  Funny you should mention that.  I was just thinking a few
> minutes ago about working on that exact change...

I thought about this a couple of weeks ago. I recalled to have read 
something about the principle: if there is more than one merge-base 
candidate, it starts by merging the merge-base candidates until there is 
only one 'virtual' merge-base candidate.

However, looking at the code I fainted. Sure, a lot should be way easier 
in C, because the functions are already there, _but_ it seemed too much 
work for one afternoon nevertheless (and I did not have more time to 
spare).

Ciao,
Dscho

Re: Importing Mozilla CVS into git

[Jon Smirl]

On 6/2/06, Pavel Roskin <proski@gnu.org> wrote:
> Dependency on Cygwin, Perl and Python is too much.  Windows is becoming
> a legacy system in some circles, and it may run on legacy hardware.  Yet
> it's irreplaceable as a testing platform for many projects.

80% of Mozilla commiters are running Windows. Some are OS bilingual
but many are not.

-- 
Jon Smirl
jonsmirl@gmail.com

Re: Importing Mozilla CVS into git

[Igor Bukanov]

On 6/2/06, Jon Smirl <jonsmirl@gmail.com> wrote:
> On 6/2/06, Pavel Roskin <proski@gnu.org> wrote:
> > Dependency on Cygwin, Perl and Python is too much.  Windows is becoming
> > a legacy system in some circles, and it may run on legacy hardware.  Yet
> > it's irreplaceable as a testing platform for many projects.
>
> 80% of Mozilla commiters are running Windows. Some are OS bilingual
> but many are not.

Mozilla build system on Windows requires Cygwin and there are 198 Perl
files in Firefox tree. So it is only Python that can be problematic.

Regards, Igor

Re: Importing Mozilla CVS into git

[Pavel Roskin]

On Wed, 2006-06-07 at 11:02 +0200, Igor Bukanov wrote:
> On 6/2/06, Jon Smirl <jonsmirl@gmail.com> wrote:
> > On 6/2/06, Pavel Roskin <proski@gnu.org> wrote:
> > > Dependency on Cygwin, Perl and Python is too much.  Windows is becoming
> > > a legacy system in some circles, and it may run on legacy hardware.  Yet
> > > it's irreplaceable as a testing platform for many projects.
> >
> > 80% of Mozilla commiters are running Windows. Some are OS bilingual
> > but many are not.
> 
> Mozilla build system on Windows requires Cygwin and there are 198 Perl
> files in Firefox tree. So it is only Python that can be problematic.

Then maybe the existing 3 python files in git (I'm not counting
compat/subprocess.py) could be converted to Perl?  Perl would be great
as the "common denominator" for interpreted languages.

Search for "python to perl" translator lead me to Perthon:
http://perthon.sourceforge.net/

But Perthon needs work.  My attempt to run it on git-p4import.py failed:

$ perl -I `pwd`/lib perthon.pl git-p4import.py 
Can't coerce array into hash at lib/Perthon/PerthonImpl.pm line 15420.

It may be a fun project for somebody who wants to learn Perl and Python.

-- 
Regards,
Pavel Roskin

Re: Importing Mozilla CVS into git

[Jon Smirl]

On 6/7/06, Igor Bukanov <igor.bukanov@gmail.com> wrote:
> On 6/2/06, Jon Smirl <jonsmirl@gmail.com> wrote:
> > On 6/2/06, Pavel Roskin <proski@gnu.org> wrote:
> > > Dependency on Cygwin, Perl and Python is too much.  Windows is becoming
> > > a legacy system in some circles, and it may run on legacy hardware.  Yet
> > > it's irreplaceable as a testing platform for many projects.
> >
> > 80% of Mozilla commiters are running Windows. Some are OS bilingual
> > but many are not.
>
> Mozilla build system on Windows requires Cygwin and there are 198 Perl
> files in Firefox tree. So it is only Python that can be problematic.

Other people have sent me mail saying this may not be as big  as
problem as was thought, only documentation people on WIndows may be an
issues.


>
> Regards, Igor
>


-- 
Jon Smirl
jonsmirl@gmail.com

Re: Importing Mozilla CVS into git

[Jakub Narebski]

Jon Smirl wrote:

> On 6/7/06, Igor Bukanov <igor.bukanov@gmail.com> wrote:
>> On 6/2/06, Jon Smirl <jonsmirl@gmail.com> wrote:
>>> On 6/2/06, Pavel Roskin <proski@gnu.org> wrote:
>>>> Dependency on Cygwin, Perl and Python is too much.  Windows is becoming
>>>> a legacy system in some circles, and it may run on legacy hardware. Yet
>>>> it's irreplaceable as a testing platform for many projects.
>>>
>>> 80% of Mozilla commiters are running Windows. Some are OS bilingual
>>> but many are not.
>>
>> Mozilla build system on Windows requires Cygwin and there are 198 Perl
>> files in Firefox tree. So it is only Python that can be problematic.
> 
> Other people have sent me mail saying this may not be as big  as
> problem as was thought, only documentation people on WIndows may be an
> issues.

With 1.4.0 there should be tar files of documentation. For now, one can use
html and man branches of git.git repository: see the INSTALL file and/or

  http://git.or.cz/gitwiki/GitDocumentation

-- 
Jakub Narebski
Warsaw, Poland

Re: Importing Mozilla CVS into git

[Linus Torvalds]

On Wed, 7 Jun 2006, Jakub Narebski wrote:
> > Other people have sent me mail saying this may not be as big  as
> > problem as was thought, only documentation people on WIndows may be an
> > issues.
> 
> With 1.4.0 there should be tar files of documentation. For now, one can use
> html and man branches of git.git repository: see the INSTALL file and/or

I think you misunderstood.

My guess is that it's the _mozilla_ documentation people that don't 
necessarily have cygwin and perl, because they don't work with the normal 
build.

Ie there are people who can write user documentation, without them having 
any clue - or caring about - build systems.

			Linus

Re: Importing Mozilla CVS into git

[Martin Langhoff]

On 6/8/06, Linus Torvalds <torvalds@osdl.org> wrote:
> My guess is that it's the _mozilla_ documentation people that don't
> necessarily have cygwin and perl, because they don't work with the normal
> build.
>
> Ie there are people who can write user documentation, without them having
> any clue - or caring about - build systems.

Which means that git-cvsserver can probably help them -- I don't think
documentation people (and translators, graphic artists, etc) need all
the git smarts. A simplistic TortoiseCVS + git-cvsserver should fit
their usage...

cheers,


martin

Re: Importing Mozilla CVS into git

[Martin Langhoff]

On 6/2/06, Jon Smirl <jonsmirl@gmail.com> wrote:
> It is going to have to be native Windows to move some of the
> developers over. They are true blue MS types that won't touch anything
> close to Unix so cygwin is out.

As others have pointed out, you have git-cvsserver which emulates a
CVS server on top of GIT, so it can be used with (almost any) CVS
client. They will be 2nd class citizens however...

cheers,


martin

Re: Importing Mozilla CVS into git

[Robin Rosenberg (list subscriber)]

fredag 02 juni 2006 06:16 skrev Martin Langhoff:
> On 6/2/06, Jon Smirl <jonsmirl@gmail.com> wrote:
> > It is going to have to be native Windows to move some of the
> > developers over. They are true blue MS types that won't touch anything
> > close to Unix so cygwin is out.
That could be fixed with nice packaging since many CVS users in Windows
never use a command line anyway since TortoiseCVS is so nice. 

> As others have pointed out, you have git-cvsserver which emulates a
> CVS server on top of GIT, so it can be used with (almost any) CVS
> client. They will be 2nd class citizens however...

(Yet) Another problem is that many windows tools use CR LF as the line ending.
Almost all windows editors default to CRLF and some detect existing line 
endings. No editing with notepad anymore. Of course that is a problem 
regardless of whether a git or cvs client is used. You'll get these big 
everything-changed commits that alter between CRLF and LF.

-- robin

Re: Importing Mozilla CVS into git

[Linus Torvalds]

On Sun, 4 Jun 2006, Robin Rosenberg (list subscriber) wrote:
> 
> (Yet) Another problem is that many windows tools use CR LF as the line ending.
> Almost all windows editors default to CRLF and some detect existing line 
> endings. No editing with notepad anymore. Of course that is a problem 
> regardless of whether a git or cvs client is used. You'll get these big 
> everything-changed commits that alter between CRLF and LF.

The only sane approach there (if you want to be at all cross-platform) is 
to just force everybody to _commit_ in UNIX '\n'-only format. Especially 
as most Windows tools probably handle that fine on reading (just have 
trouble writing them).

And that shouldn't actually be that hard to do. The most trivial approach 
is to have just a pre-trigger on commits, but let's face it, that would 
not be a good "full" solution. A better one is to just make the whole
"git update-index" thing just have a "automatically ignore CR/LF" mode.

Which really shouldn't be that hard. I think it's literally a matter of 
teaching "index_fd()" in sha1_file.c to recognize text-files, and remove 
CR/LF from them. All done (except to add the flag that enables the 
detection, of course - just so that sane systems won't have the overhead 
or the "corrupt binary files" issue).

Something like this is TOTALLY UNTESTED!

(You also need to teach "diff" to ignore differences in cr/lf, and this 
patch is bad because it's unconditional, and probably doesn't work 
anyway, but hey, the idea is possibly sound. Maybe)

		Linus
---
diff --git a/sha1_file.c b/sha1_file.c
index aea0f40..6dc6a3f 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -1740,9 +1740,30 @@ int index_pipe(unsigned char *sha1, int 
 	return ret;
 }
 
+static unsigned long autodetect_crlf(unsigned char *src, unsigned long size)
+{
+	unsigned long newsize = 0;
+	unsigned char *dst = src;
+	unsigned char last = 0;
+
+	while (size) {
+		unsigned char c = *src++;
+		if (last == '\r' && c == '\n') {
+			dst[-1] = '\n';
+		} else {
+			newsize++;
+			dst++;
+			if (dst != src)
+				dst[-1] = c;
+		}
+		last = c;
+	}
+	return newsize;
+}
+
 int index_fd(unsigned char *sha1, int fd, struct stat *st, int write_object, const char *type)
 {
-	unsigned long size = st->st_size;
+	unsigned long size = st->st_size, use_size;
 	void *buf;
 	int ret;
 	unsigned char hdr[50];
@@ -1755,12 +1776,15 @@ int index_fd(unsigned char *sha1, int fd
 	if (buf == MAP_FAILED)
 		return -1;
 
-	if (!type)
+	use_size = size;
+	if (!type) {
 		type = blob_type;
+		use_size = autodetect_crlf(buf, size);
+	}
 	if (write_object)
-		ret = write_sha1_file(buf, size, type, sha1);
+		ret = write_sha1_file(buf, use_size, type, sha1);
 	else {
-		write_sha1_file_prepare(buf, size, type, sha1, hdr, &hdrlen);
+		write_sha1_file_prepare(buf, use_size, type, sha1, hdr, &hdrlen);
 		ret = 0;
 	}
 	if (size)

Re: Importing Mozilla CVS into git

[Bertrand Jacquin]

On 6/4/06, Linus Torvalds <torvalds@osdl.org> wrote:
>
>
> On Sun, 4 Jun 2006, Robin Rosenberg (list subscriber) wrote:
> >
> > (Yet) Another problem is that many windows tools use CR LF as the line ending.
> > Almost all windows editors default to CRLF and some detect existing line
> > endings. No editing with notepad anymore. Of course that is a problem
> > regardless of whether a git or cvs client is used. You'll get these big
> > everything-changed commits that alter between CRLF and LF.
>
> The only sane approach there (if you want to be at all cross-platform) is
> to just force everybody to _commit_ in UNIX '\n'-only format. Especially
> as most Windows tools probably handle that fine on reading (just have
> trouble writing them).
>
> And that shouldn't actually be that hard to do. The most trivial approach
> is to have just a pre-trigger on commits, but let's face it, that would
> not be a good "full" solution. A better one is to just make the whole
> "git update-index" thing just have a "automatically ignore CR/LF" mode.
>
> Which really shouldn't be that hard. I think it's literally a matter of
> teaching "index_fd()" in sha1_file.c to recognize text-files, and remove
> CR/LF from them. All done (except to add the flag that enables the
> detection, of course - just so that sane systems won't have the overhead
> or the "corrupt binary files" issue).
>
> Something like this is TOTALLY UNTESTED!
>
> (You also need to teach "diff" to ignore differences in cr/lf, and this
> patch is bad because it's unconditional, and probably doesn't work
> anyway, but hey, the idea is possibly sound. Maybe)

Is it also apply for binary files ? It could corrupt files as well.
If end-user application don't understand '\n' but '\r\n', you can have
bad issues (I think to notepad here (yes crappy, but ..)). Couldn't it
be configurable ?

-- 
# Beber : beber@gna.org
# IM : beber@jabber.fr
# http://guybrush.ath.cx, irc://irc.freenode.net/#{e.fr,gentoofr}

Re: Importing Mozilla CVS into git

[Jakub Narebski]

On Sun, 4 Jun 2006, Linus Torvalds wrote:

> On Sun, 4 Jun 2006, Robin Rosenberg (list subscriber) wrote:
>> 
>> (Yet) Another problem is that many windows tools use CR LF as the line ending.
>> Almost all windows editors default to CRLF and some detect existing line 
>> endings. No editing with notepad anymore. Of course that is a problem 
>> regardless of whether a git or cvs client is used. You'll get these big 
>> everything-changed commits that alter between CRLF and LF.
> 
> The only sane approach there (if you want to be at all cross-platform) is 
> to just force everybody to _commit_ in UNIX '\n'-only format. Especially 
> as most Windows tools probably handle that fine on reading (just have 
> trouble writing them).
> 
> And that shouldn't actually be that hard to do. The most trivial approach 
> is to have just a pre-trigger on commits, but let's face it, that would 
> not be a good "full" solution. A better one is to just make the whole
> "git update-index" thing just have a "automatically ignore CR/LF" mode.

Why wouldn't it be good solution?

BTW. wouldn't Mercurial encode/decode filters

  http://www.selenic.com/mercurial/wiki/index.cgi/EncodeDecodeFilter

be a better solution than modifying files by "git update-index", 
with all problems it can cause (not detected binary files, text files
which have to be in CR/LF line ending,...).

-- 
Jakub Narebski
Warsaw, Poland

Re: Importing Mozilla CVS into git

[Linus Torvalds]

On Sun, 4 Jun 2006, Jakub Narebski wrote:
> > 
> > And that shouldn't actually be that hard to do. The most trivial approach 
> > is to have just a pre-trigger on commits, but let's face it, that would 
> > not be a good "full" solution. A better one is to just make the whole
> > "git update-index" thing just have a "automatically ignore CR/LF" mode.
> 
> Why wouldn't it be good solution?

The pre-commit filter thing should work fine, and hey, maybe it's worth 
doing that way. I just worry/think that it will result in tons of noise 
when you do a "git diff" and "git update-index --refresh" on a file that 
has been changed, but then the change reverted.

But I didn't really think it through very deeply, it was just an idle "I 
think the pre-commit hook will fall down when X happens that is a 
non-commit event" thought. I suspect this is one of those things where 
somebody actually working in that kind of environment will figure out what 
the problems are, and what the righ solution is.

> BTW. wouldn't Mercurial encode/decode filters
> 
>   http://www.selenic.com/mercurial/wiki/index.cgi/EncodeDecodeFilter
> 
> be a better solution than modifying files by "git update-index", 
> with all problems it can cause (not detected binary files, text files
> which have to be in CR/LF line ending,...).

Please do realize that the patch I sent out was absolutely _not_ meant to 
be taken seriously. It was more a "somebody could try this in a windows 
environment, and if it works as an approach, we can try to do it right".

I'm absolutely _not_ suggesting merging that patch as-is or even in any 
form very close to it. It clearly needs a config file entry with filename 
patterns etc at a minimum.

		Linus

Re: Importing Mozilla CVS into git

[Robin Rosenberg (list subscriber)]

söndag 04 juni 2006 19:55 skrev Linus Torvalds:
> On Sun, 4 Jun 2006, Jakub Narebski wrote:
> > > And that shouldn't actually be that hard to do. The most trivial
> > > approach is to have just a pre-trigger on commits, but let's face it,
> > > that would not be a good "full" solution. A better one is to just make
> > > the whole "git update-index" thing just have a "automatically ignore
> > > CR/LF" mode.
> >
> > Why wouldn't it be good solution?
>
> The pre-commit filter thing should work fine, and hey, maybe it's worth
> doing that way. I just worry/think that it will result in tons of noise
> when you do a "git diff" and "git update-index --refresh" on a file that
> has been changed, but then the change reverted.
>
> But I didn't really think it through very deeply, it was just an idle "I
> think the pre-commit hook will fall down when X happens that is a
> non-commit event" thought. I suspect this is one of those things where
> somebody actually working in that kind of environment will figure out what
> the problems are, and what the righ solution is.
>
> > BTW. wouldn't Mercurial encode/decode filters
> >
> >   http://www.selenic.com/mercurial/wiki/index.cgi/EncodeDecodeFilter
> >
> > be a better solution than modifying files by "git update-index",
> > with all problems it can cause (not detected binary files, text files
> > which have to be in CR/LF line ending,...).
>
> Please do realize that the patch I sent out was absolutely _not_ meant to
> be taken seriously. It was more a "somebody could try this in a windows
> environment, and if it works as an approach, we can try to do it right".

Other version control systems simply treat text and binary files differently. 
No smart(ass) logic doing the wrong thing. A text file gets processed on
check-in AND checkout depending on it's type and the client setting.
Some heuristics may be applied when adding files. i.e look-up according to 
magic cookies or looking for bytes that simply do not occur in text files 
(e..g a nul byte).  Those few systems that I know about treat the type as a 
file (as opposed to a version specific) attribute. Some systems have lots of 
file types, not just text and binary. Encoding is about the only thing that 
would interest me, although not terribly important (except the file name), 
but that may be off topic for this thread.

The hash-on-the whole-tree might be a reason for making the attribute 
version-specific.

Mercurial's filters sounds like a good way to implement file types in a 
generic way as long as git's excellent performance isn't hurt.

> I'm absolutely _not_ suggesting merging that patch as-is or even in any
> form very close to it. It clearly needs a config file entry with filename
> patterns etc at a minimum.

Do people apply your patches right away, like it's some god-like commandments?

-- robin

Re: Importing Mozilla CVS into git

[Linus Torvalds]

On Sun, 4 Jun 2006, Robin Rosenberg (list subscriber) wrote:
> 
> Other version control systems simply treat text and binary files differently. 
> No smart(ass) logic doing the wrong thing.

Treating text and binary file differently _is_ the "smart(ass) logic doing 
the wrong thing".

Git really shouldn't do that. The patch was meant to show how you really 
don't need to - the internal objects would never be "binary vs text", 
there would be a way to just basically map one onto another.

> > I'm absolutely _not_ suggesting merging that patch as-is or even in any
> > form very close to it. It clearly needs a config file entry with filename
> > patterns etc at a minimum.
> 
> Do people apply your patches right away, like it's some god-like commandments?

What's your problem here, exactly? 

I was just trying to point out that my patch was an example, where 
somebody who cares (not me) can use it as a starting point.

If you can't be civil, at least be quiet, ok?

		Linus

Re: Importing Mozilla CVS into git

[Robin Rosenberg (list subscriber)]

söndag 04 juni 2006 22:00 skrev Linus Torvalds:
> On Sun, 4 Jun 2006, Robin Rosenberg (list subscriber) wrote:
> > Other version control systems simply treat text and binary files
> > differently. No smart(ass) logic doing the wrong thing.
>
> Treating text and binary file differently _is_ the "smart(ass) logic doing
> the wrong thing".
>
> Git really shouldn't do that. The patch was meant to show how you really
> don't need to - the internal objects would never be "binary vs text",
> there would be a way to just basically map one onto another.

Your patch assumes all files are text and the transformation doesn't corrupt 
the file, which isn't true. CR-LF combinations cannot be translated to LF and
vice verse in all files, simply becuase what looks like CR LF isn't two 
characters, but something else. Looking for a nul byte and possibly some
other magic byte would make you right more often, but not always.

[...]
> If you can't be civil, at least be quiet, ok?
>
> 		Linus
A bad joke, I'm sorry. It wasn't ment to be offensive.

-- robin

Re: Importing Mozilla CVS into git

[Robin Rosenberg (list subscriber)]

Just forget this post... @_@ (except the last line)

söndag 04 juni 2006 23:25 skrev Robin Rosenberg (list subscriber):
> söndag 04 juni 2006 22:00 skrev Linus Torvalds:
> > On Sun, 4 Jun 2006, Robin Rosenberg (list subscriber) wrote:
> > > Other version control systems simply treat text and binary files
> > > differently. No smart(ass) logic doing the wrong thing.
> >
> > Treating text and binary file differently _is_ the "smart(ass) logic
> > doing the wrong thing".
> >
> > Git really shouldn't do that. The patch was meant to show how you really
> > don't need to - the internal objects would never be "binary vs text",
> > there would be a way to just basically map one onto another.
>
> Your patch assumes all files are text and the transformation doesn't
> corrupt the file, which isn't true. CR-LF combinations cannot be translated
> to LF and vice verse in all files, simply becuase what looks like CR LF
> isn't two characters, but something else. Looking for a nul byte and
> possibly some other magic byte would make you right more often, but not
> always.
>
> [...]
>
> > If you can't be civil, at least be quiet, ok?
> >
> > 		Linus
>
> A bad joke, I'm sorry. It wasn't ment to be offensive.
>

Re: Importing Mozilla CVS into git

[Linus Torvalds]

On Sun, 4 Jun 2006, Robin Rosenberg (list subscriber) wrote:
>
> Your patch assumes all files are text and the transformation doesn't corrupt 
> the file, which isn't true.

How do you think things get done? You test the _technology_ first, and 
then if that is shown to be workable in a real environment, _then_ do you 
actually add the polish to make it useful.

That was all the patch was. A technology demonstration. I'd really like to 
hear whether it works in a simple CR/LF environment, because if it 
doesn't, then it needs some totally different approach.

And yes, I could test it myself, but (a) I'm way too lazy and (b) I 
consciously try to get others involved because it's a better long-term 
strategy (because I expect to be ay too lazy in the future too)

		Linus

Re: Importing Mozilla CVS into git

[Yakov Lerner]

On 6/4/06, Linus Torvalds <torvalds@osdl.org> wrote:
+static unsigned long autodetect_crlf(unsigned char *src, unsigned long size)
> +{
> +       unsigned long newsize = 0;
> +       unsigned char *dst = src;
> +       unsigned char last = 0;
> +
> +       while (size) {
> +               unsigned char c = *src++;

size--;
is missing

> +               if (last == '\r' && c == '\n') {
> +                       dst[-1] = '\n';
> +               } else {
> +                       newsize++;
> +                       dst++;
> +                       if (dst != src)
> +                               dst[-1] = c;
> +               }
> +               last = c;
> +       }
> +       return newsize;
> +}

Yakov

Re: Importing Mozilla CVS into git

[Jon Smirl]

If I parsecvs this pair it fails:

/home/jonsmirl/mozilla/mozilla/dom/src/base/nsFocusController.h,v
/home/jonsmirl/mozilla/mozilla/dom/src/base/nsGlobalWindow.cpp,v

[jonsmirl@jonsmirl foo]$ ../parsecvs <sm2
defaulting to local storage area
Load:                nsGlobalWindow.cpp,v ....................*     2 of     2
Warning: branch point MOZILLA_1_0_BRANCH -> master matched by date
fatal: Not a valid object name (null)
git-read-tree '(null)' failed
[jonsmirl@jonsmirl foo]$

But parsecvs works on each of them individually.

-- 
Jon Smirl
jonsmirl@gmail.com

Re: Importing Mozilla CVS into git

[Jon Smirl]

On 6/1/06, Jon Smirl <jonsmirl@gmail.com> wrote:
> With the attached patch you can parse the entire Mozilla tree. The
> tree has over 100,000 files in it and about 300 branches.

I was a little low with these counts, more like 110,000 files and some
parts of the tree have 1,000 branches. Total tree size is 3GB.

-- 
Jon Smirl
jonsmirl@gmail.com

Re: Importing Mozilla CVS into git

[Martin Langhoff]

On 6/3/06, Jon Smirl <jonsmirl@gmail.com> wrote:
> On 6/1/06, Jon Smirl <jonsmirl@gmail.com> wrote:
> > With the attached patch you can parse the entire Mozilla tree. The
> > tree has over 100,000 files in it and about 300 branches.
>
> I was a little low with these counts, more like 110,000 files and some
> parts of the tree have 1,000 branches. Total tree size is 3GB.

I don't think it really has that many branches. If I am to believe
cvsps (which took 3GB to walk the history), it has some branches with
recursive loops in their ancestry (MANG_MATH_BRANCH and
SpiderMonkey140_BRANCH have eachother as ancestors!?), 197969 commits
and 796 branches.

This repository has been mangled quite badly. Don't know what you guys
did with it, but it sure isn't pretty. I'm working on getting
git-cvsimport to get through a complete import.

cheers,



martin

Re: Importing Mozilla CVS into git

[Jon Smirl]

On 6/6/06, Martin Langhoff <martin.langhoff@gmail.com> wrote:
> On 6/3/06, Jon Smirl <jonsmirl@gmail.com> wrote:
> > On 6/1/06, Jon Smirl <jonsmirl@gmail.com> wrote:
> > > With the attached patch you can parse the entire Mozilla tree. The
> > > tree has over 100,000 files in it and about 300 branches.
> >
> > I was a little low with these counts, more like 110,000 files and some
> > parts of the tree have 1,000 branches. Total tree size is 3GB.
>
> I don't think it really has that many branches. If I am to believe
> cvsps (which took 3GB to walk the history), it has some branches with
> recursive loops in their ancestry (MANG_MATH_BRANCH and
> SpiderMonkey140_BRANCH have eachother as ancestors!?), 197969 commits
> and 796 branches.

It probably is 796 and not a 1,000. The branch names were scrolling
across my screen and I just estimated.

> This repository has been mangled quite badly. Don't know what you guys
> did with it, but it sure isn't pretty. I'm working on getting
> git-cvsimport to get through a complete import.

The repository is close to 10 years old and it has gone through a
number of corporate reorgs. Who knows what has happened to it over
that length of time.

Have you looked at the SVN CVS import tool? It imported Mozilla on the
first try. If you download the source they have built about 40 test
repositories with various errors. Those would make a good test suite
for cvsps.  http://cvs2svn.tigris.org

I have been working on converting the svn tool to do git commands but
my git knowledge is limited so it has been slow going. The last stage,
pass 8, is very similar to what the git tools do. The svn commands
just need to be swapped for git ones.

If you get git-cvsimport working I'll use it instead. Will the cvsps
process stay small enough to run on a 32b machine? The svn tools are
very RAM efficient since they use an external db. Can cvsps read from
a local copy of the repository without using a CVS server?

We are going to have to develop some kind of incremental mechanism for
updating the new git tree. It can take up to two days to convert the
repository, Mozilla development can't be shut down that long for a
transition. Git will also need to mirror the CVS repository (check-in
still going to CVS) for a long time while we convince everyone on the
merits of switching.

My imported svn version of Mozilla has a lot of performance problems.
One of the directories has over 200,000 files in it slowing downing
the filesystem. The repository went from 3GB CVS to 8GB svn, probably
due to svn using 1000s of tiny files. I'll look around and see if svn
has a pack feature like git.

-- 
Jon Smirl
jonsmirl@gmail.com

Re: Importing Mozilla CVS into git

[Martin Langhoff]

On 6/7/06, Jon Smirl <jonsmirl@gmail.com> wrote:
> Have you looked at the SVN CVS import tool? It imported Mozilla on the
> first try. If you download the source they have built about 40 test
> repositories with various errors. Those would make a good test suite
> for cvsps.  http://cvs2svn.tigris.org

Haven't yet, but I'll do. I'm currently quite busy at work, but I'm
running these imports (Moz, Gentoo) and trying to address issues
arising. Will look into SVN's tool it when I have a bit more time.

What I'll probably do is steal those test cases! ;-)

> I have been working on converting the svn tool to do git commands but
> my git knowledge is limited so it has been slow going. The last stage,
> pass 8, is very similar to what the git tools do. The svn commands
> just need to be swapped for git ones.

That would be interesting. And yet, I would have to evaluate how to
transition gateways running git-cvsimport incrementally to a different
importer.

Does it do incremental imports?

> If you get git-cvsimport working I'll use it instead. Will the cvsps
> process stay small enough to run on a 32b machine? The svn tools are

Currently not, but I do hope that the moz team has access to at least
one machine with more than 32MB ;-)

With the current code, you will want a 3GB machine to run the git-cvsimport

git-cvsimport has a memory leak that I've been chasing for a while and
I'll eventually fix, so it should fit in 32MB comfortably. cvsps is
memory bound, and will probably take quite a bit of work to fix that.
However, I suspect we can make it a lot more efficient.

> very RAM efficient since they use an external db. Can cvsps read from
> a local copy of the repository without using a CVS server?

> We are going to have to develop some kind of incremental mechanism for
> updating the new git tree. It can take up to two days to convert the
> repository, Mozilla development can't be shut down that long for a

You don't have to. Run an initial import, and then freeze development
and run an incremental -- which will take an hour at the most. And
then your mozilla.git repo is ready and up to date.

> transition. Git will also need to mirror the CVS repository (check-in
> still going to CVS) for a long time while we convince everyone on the
> merits of switching.

That's easy -- run git-cvsimport on a cronjob.

> My imported svn version of Mozilla has a lot of performance problems.
> One of the directories has over 200,000 files in it slowing downing
> the filesystem. The repository went from 3GB CVS to 8GB svn, probably
> due to svn using 1000s of tiny files. I'll look around and see if svn
> has a pack feature like git.

At least they got a good importer ;-)

cheers,


martin

Re: Importing Mozilla CVS into git

[Keith Packard]

[-- Attachment #1: Type: text/plain, Size: 993 bytes --]

On Wed, 2006-06-07 at 07:57 +1200, Martin Langhoff wrote:

> git-cvsimport has a memory leak that I've been chasing for a while and
> I'll eventually fix, so it should fit in 32MB comfortably. cvsps is
> memory bound, and will probably take quite a bit of work to fix that.
> However, I suspect we can make it a lot more efficient.

Yeah, parsecvs is a memory pig as well -- it builds a giant in-memory
representation of the entire project history using flat lists of files
for every revision, just like git used to do. Fixing that should make it
run in small amounts of memory; it only needs 40 bytes per file revision
for the raw data as it converts the cvs files to git objects as it reads
them, saving only the hash value in memory.

Not relying on cvsps has been a huge feature though; cvsps loses a
tremendous amount of data, along with making several gross and difficult
to fix errors in project history from several of my repositories.

-- 
keith.packard@intel.com

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: Importing Mozilla CVS into git

[Jon Smirl]

On 6/6/06, Jon Smirl <jonsmirl@gmail.com> wrote:
> On 6/6/06, Martin Langhoff <martin.langhoff@gmail.com> wrote:
> > On 6/3/06, Jon Smirl <jonsmirl@gmail.com> wrote:
> > > On 6/1/06, Jon Smirl <jonsmirl@gmail.com> wrote:
> > > > With the attached patch you can parse the entire Mozilla tree. The
> > > > tree has over 100,000 files in it and about 300 branches.
> > >
> > > I was a little low with these counts, more like 110,000 files and some
> > > parts of the tree have 1,000 branches. Total tree size is 3GB.
> >
> > I don't think it really has that many branches. If I am to believe
> > cvsps (which took 3GB to walk the history), it has some branches with
> > recursive loops in their ancestry (MANG_MATH_BRANCH and
> > SpiderMonkey140_BRANCH have eachother as ancestors!?), 197969 commits
> > and 796 branches.

My full import to svn just finished after a day and a half.
Here are the stats:

cvs2svn Statistics:
------------------
Total CVS Files:             99851
Total CVS Revisions:        948580
Total Unique Tags:            1505
Total Unique Branches:        1577
CVS Repos Size in KB:      2725843
Total SVN Commits:          205787
First Revision Date:    Fri Mar 27 21:13:08 1998
Last Revision Date:     Tue May 30 19:28:10 2006
------------------
Timings:
------------------
pass 1:  3602 seconds
pass 2:   227 seconds
pass 3:    66 seconds
pass 4:  1070 seconds
pass 8:124650 seconds
total: 124650 seconds
[jonsmirl@jonsmirl ~]$

[jonsmirl@jonsmirl svn]$ du -h
4.0K    ./svntest/dav
12K     ./svntest/locks
40K     ./svntest/hooks
16K     ./svntest/conf
7.4G    ./svntest/db/revs
808M    ./svntest/db/revprops
4.0K    ./svntest/db/transactions
8.2G    ./svntest/db
8.2G    ./svntest
8.2G    .

[jonsmirl@jonsmirl svn]$ find | wc
 411607  411607 10891057

There are two directories that each contain about 205k files. 205K
files in a single directory is causing svn problems on Ext3.

Bottom line, cvs2svn import tool works quite well. Highest memory
consumption I saw was 100MB and it used 6GB of extra disk while
running plus space need by svn.

I don't know quite enough about git yet to replace the svn commands it
uses with git equivalents but if that were done I think most of the
cvs import problems would be solved. Obviously the svn team has put a
great deal of work into this program.

I don't think replacing the svn commands is very hard, I just haven't
figured out the right way to build branches with low-level git yet and
I don't know Python. I'll bet someone already familiar with git and
cvs import could convert it in a couple of hours.

-- 
Jon Smirl
jonsmirl@gmail.com

Re: Importing Mozilla CVS into git

[Linus Torvalds]

On Thu, 1 Jun 2006, Jon Smirl wrote:
>
> I've been working on importing the Mozilla CVS into git for the last
> few days. I've fixed up parsecvs so that it can parse the entire
> repository without errors. Now I'm running into problems because there
> are over 300 branches.
> 
> I just run into a problem with git show-branch. Mozilla CVS has a lot
> more than 29 refs, is this something that can be expanded?

Hmm.. Any reason you care about "show-branch --all" in particular?

The algorithm used for show-branch really doesn't scale well, it needs one 
bit per commit per branch, and I didn't realize anybody could ever really 
care.

		Linus

Re: Importing Mozilla CVS into git

[Jon Smirl]

On 6/1/06, Linus Torvalds <torvalds@osdl.org> wrote:
>
>
> On Thu, 1 Jun 2006, Jon Smirl wrote:
> >
> > I've been working on importing the Mozilla CVS into git for the last
> > few days. I've fixed up parsecvs so that it can parse the entire
> > repository without errors. Now I'm running into problems because there
> > are over 300 branches.
> >
> > I just run into a problem with git show-branch. Mozilla CVS has a lot
> > more than 29 refs, is this something that can be expanded?
>
> Hmm.. Any reason you care about "show-branch --all" in particular?
>
> The algorithm used for show-branch really doesn't scale well, it needs one
> bit per commit per branch, and I didn't realize anybody could ever really
> care.

I was trying to use it to figure out what was wrong with the branch
processing in parsecvs. It doesn't have to be fixed. show-branch --all
fails with same 29 tag limit.

-- 
Jon Smirl
jonsmirl@gmail.com

Re: Importing Mozilla CVS into git

[Linus Torvalds]

On Thu, 1 Jun 2006, Jon Smirl wrote:
> 
> I was trying to use it to figure out what was wrong with the branch
> processing in parsecvs. It doesn't have to be fixed. show-branch --all
> fails with same 29 tag limit.

You're much better off using "gitk --all" if you want to see the result, 
the "show-branch" this is really broken. It is using the old algorithm 
that we used to use for "git-rev-tree", and got rid of about a year ago 
there in favour of git-rev-list ;)

			Linus

Re: Importing Mozilla CVS into git

[Junio C Hamano]

Linus Torvalds <torvalds@osdl.org> writes:

> You're much better off using "gitk --all" if you want to see the result, 
> the "show-branch" this is really broken. It is using the old algorithm 
> that we used to use for "git-rev-tree", and got rid of about a year ago 
> there in favour of git-rev-list ;)

Are you sure about it?  My recollection is it uses the
merge-base logic, naturally enhanced for multiple heads.

And enhancing it to support more than one int wide bitmap should
not be too difficult, although looking at the output would be
very taxing for human eye, so I do not know if it is worth it.

Re: Importing Mozilla CVS into git

[Linus Torvalds]

On Thu, 1 Jun 2006, Junio C Hamano wrote:
> > You're much better off using "gitk --all" if you want to see the result, 
> > the "show-branch" this is really broken. It is using the old algorithm 
> > that we used to use for "git-rev-tree", and got rid of about a year ago 
> > there in favour of git-rev-list ;)
> 
> Are you sure about it?  My recollection is it uses the
> merge-base logic, naturally enhanced for multiple heads.

Well, it's all the same algorithm, where just the bit usage differs. 
git-rev-tree is slightly closer, if only because the original 
git-merge-base only did two heads, if I recall correctly (while 
git-rev-tree did 16 - the ability of git-show-branch to do 29 came from 
just using all the free bits rather than the high bits like rev-tree did)

> And enhancing it to support more than one int wide bitmap should
> not be too difficult, although looking at the output would be
> very taxing for human eye, so I do not know if it is worth it.

Yeah, I don't think there is any reason to really support it. If you have 
more than a few heads, you really do need the graphical version to see 
what is going on, and git-show-branch doesn't buy you anything.

		Linus

Re: Importing Mozilla CVS into git

[Junio C Hamano]

Linus Torvalds <torvalds@osdl.org> writes:

> Yeah, I don't think there is any reason to really support it. If you have 
> more than a few heads, you really do need the graphical version to see 
> what is going on, and git-show-branch doesn't buy you anything.

The real reason it uses a bit per given ref is what it wants to
show is different from what gitk shows.  It wants to show which
ones are reachable from which head on each commit -- in gitk the
user has to follow the line to find it out.

However, to track 300 branches, you would need a terminal with
360 columns or so, _and_ you have to count columns to see if a
given commit is reachable from the ref you are interested in, so
it is not useful at all in practice to do more than a handful
refs at a time.

Re: Importing Mozilla CVS into git

[Martin Langhoff]

Jon,

> Is anyone interested in helping out with this? My knowledge of git and
> CVS is limited. Mozilla CVS is about 3GB and it is available via
> rsync. I can post the parsecvs changes if wanted.

Fetchin it now, I'll definitely have a play. Have you tried with a
recent git-cvsimport? In the last 2 weeks it's seen a lot of
performance & scalability improvements as we were importing the
gentoo-x86 tree.

Grab the latest 'master' branch from Junio and give the import a go.

cheers,



martin