synchronizing incremental git changes to cvs

synchronizing incremental git changes to cvs

[Jim Meyering]

Hello,

Can anyone point me at code to mirror a git repository to cvs?
I'd like to develop using git, and have a commit hook mirror the
day-to-day changes (tags/commits) made in the git repo to a
cvs repository.  The idea is that the only way changes get into
the cvs repo is via the git commit hook.

I've experimented with git-cvsexportcommit, and found a few bugs (it
couldn't handle simple things, like adding a file in a new directory --
fixed that, along with a few other minor problems), adding an empty file
in git still gets a patch application error on the cvs side, but I can
live with that for now.  More seriously, making a change on a git branch
mistakenly tries to apply the delta on the cvs trunk.  None of this is
particularly hard to fix -- or even critical, as long as you don't care
about branches.  I'm just hoping someone has already produced something
more robust.  From the looks of darcs/tailor, it doesn't handle the use
of git as a source.

Why am I interested?  I want to switch the development of GNU coreutils
from cvs to git.  I would also like to continue making the repository
available via cvs, for the sake of continuity.  At worst, I can always
cut the CVS cord, but that's a last resort.

Jim

Re: synchronizing incremental git changes to cvs

[Johannes Schindelin]

Hi,

On Sun, 21 May 2006, Jim Meyering wrote:

> Why am I interested?  I want to switch the development of GNU coreutils
> from cvs to git.  I would also like to continue making the repository
> available via cvs, for the sake of continuity.  At worst, I can always
> cut the CVS cord, but that's a last resort.

If you only want to make a cvs repository available for tracking the 
project, git-cvsserver is what you want. It is even faster than the 
original cvs...

Ciao,
Dscho

Re: synchronizing incremental git changes to cvs

[Jim Meyering]

Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:
> On Sun, 21 May 2006, Jim Meyering wrote:
>
>> Why am I interested?  I want to switch the development of GNU coreutils
>> from cvs to git.  I would also like to continue making the repository
>> available via cvs, for the sake of continuity.  At worst, I can always
>> cut the CVS cord, but that's a last resort.
>
> If you only want to make a cvs repository available for tracking the
> project, git-cvsserver is what you want. It is even faster than the
> original cvs...

That might work if I had sufficient access to the system hosting the
public CVS repository.  But there are restrictions (like no ssh access).
Currently I rsync the master repo to an intermediate site, from which
it is periodically pulled by savannah.  Paranoia on both sides.

If I end up leaving savannah, can someone propose a good site,
i.e., secure, yet with git and rsync access?

I haven't made the leap to git yet, but git-cvsimport (from git-1.3.2)
seems to do a very good job of converting the cvs module (89MB).

FYI, here are some stats on the resulting git repository:

Size (nothing repacked):
  1051MB (du -sh, actual, on reiserfs 3)
   708MB (du --apparent-size)

Size repacked, (via git-repack -a -d && git-prune-packed)
    65MB (du --si -s)

20k+ patchsets (counted by cvsps)
40k+ revisions (counted by cvs ... rlog cu|grep -c '^revision')

While repacking, git said something about more than 100K objects.
There were 120K files under .git/ before repacking.

Re: synchronizing incremental git changes to cvs

[Pavel Roskin]

Hello, Jim!

On Sun, 2006-05-21 at 15:40 +0200, Jim Meyering wrote:
> Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:
> > On Sun, 21 May 2006, Jim Meyering wrote:
> >
> >> Why am I interested?  I want to switch the development of GNU coreutils
> >> from cvs to git.

I believe you have a very good reason to talk to decision makers in FSF.
Savannah is very poorly maintained, and I actually took one of my
projects (Orinoco driver) to SourceForge Subversion.

If losing a Linux driver is next to nothing, losing GNU coreutils is a
big deal for the GNU development site.  You are likely to be heard if
you request git support.

>   I would also like to continue making the repository
> >> available via cvs, for the sake of continuity.  At worst, I can always
> >> cut the CVS cord, but that's a last resort.

Subversion is as easy as CVS for potential users, but it has a useful
"log" command if nothing else.  It also have real changesets, which
means no more guesswork when moving changes back and forth.

> > If you only want to make a cvs repository available for tracking the
> > project, git-cvsserver is what you want. It is even faster than the
> > original cvs...
> 
> That might work if I had sufficient access to the system hosting the
> public CVS repository.  But there are restrictions (like no ssh access).
> Currently I rsync the master repo to an intermediate site, from which
> it is periodically pulled by savannah.  Paranoia on both sides.
> 
> If I end up leaving savannah, can someone propose a good site,
> i.e., secure, yet with git and rsync access?

Sorry, I don't know any free git hosters, but here's what you can do:

1) Pressure Savannah to support git
2) Use arch on Savannah
3) Move to Subversion on SourceForge, GNA.org or Berlios and use git-svn

-- 
Regards,
Pavel Roskin

Re: synchronizing incremental git changes to cvs

[Jakub Narebski]

Pavel Roskin wrote:

> Sorry, I don't know any free git hosters, but here's what you can do:
> 
> 1) Pressure Savannah to support git
> 2) Use arch on Savannah
> 3) Move to Subversion on SourceForge, GNA.org or Berlios and use git-svn

I always thought that both Gna! and Savannah use Savane (SourceForge engine
fork), which has CVS and Subversion support, and from what I can see also
GNU Arch support. I'm quite sure that at Gna! you can choose between CVS,
SVN and Arch.

-- 
Jakub Narebski
Warsaw, Poland

Re: synchronizing incremental git changes to cvs

[Martin Langhoff]

On 5/21/06, Jim Meyering <jim@meyering.net> wrote:
> Can anyone point me at code to mirror a git repository to cvs?

Ive thought a couple of times about writing an exporter that would
replay things into a true CVS repo, but it's truly not worth it. We've
already got git-cvsserver that does all that -- better for me to focus
on that codebase.

> I've experimented with git-cvsexportcommit, and found a few bugs (it
> couldn't handle simple things, like adding a file in a new directory --
> fixed that, along with a few other minor problems), adding an empty file
> in git still gets a patch application error on the cvs side, but I can
> live with that for now.  More seriously, making a change on a git branch
> mistakenly tries to apply the delta on the cvs trunk.

cvsexportcommit is clearly for manual usage, not for automagic usage.
It is a bit rough, (and I'd like to see your patches to it!) but it
wants to be driven by a smarter script to, for instance, know what
branch you want things in.

> Why am I interested?  I want to switch the development of GNU coreutils
> from cvs to git.  I would also like to continue making the repository
> available via cvs, for the sake of continuity.  At worst, I can always
> cut the CVS cord, but that's a last resort.

git-cvsserver is the word. It currently tracks the git repo itself
pretty well (perfectly, AFAICS) and it also tracks a git tree that is
actually imported daily from CVS -- doing

    CVSrepo ->cvsimport -> GIT -> cvsserver -> CVS checkout

git-cvsserver works great for anon cvs access (does pserver) and
TortoiseCVS and cli cvs work great with it. Eclipse works well, but it
has been quite hard to get 'right'. Optionally, it can support users
with commit rights via ssh. It does track git 'heads' but they don't
show up as branches, they show up as different modules. So you to get
a checkout of the master branch, you do:

    cvs -d pserver:anonymouys@foo.com:/var/foo.git co master

hope that helps!




martin

Re: synchronizing incremental git changes to cvs

[Jim Meyering]

"Martin Langhoff" <martin.langhoff@gmail.com> wrote:
> git-cvsserver is the word. It currently tracks the git repo itself
> pretty well (perfectly, AFAICS) and it also tracks a git tree that is
> actually imported daily from CVS -- doing
>
>    CVSrepo ->cvsimport -> GIT -> cvsserver -> CVS checkout

Thanks, but I'd rather do primary development directly using git,
rather than with CVS.

Re: synchronizing incremental git changes to cvs

[Junio C Hamano]

Jim Meyering <jim@meyering.net> writes:

> I'd like to develop using git, and have a commit hook mirror the
> day-to-day changes (tags/commits) made in the git repo to a
> cvs repository.  The idea is that the only way changes get into
> the cvs repo is via the git commit hook.

I do not use the automated tools myself, but I sync the day-job
work in my git repository to CVS at work.  I do not develop with
CVS but use it merely as a publishing medium.  Although other
people can make commits into CVS in which case I have to slurp
the change back into my git repository.

 (0) Bootstrap.  I did use git-cvsimport myself (this repository
     started before the tool was written).  Instead:

     . cvs checkout the tip of the CVS development history

     . "git init-db", edit .gitignore to ignore CVS, and "git
       add ."

     . "git commit -m epoch"; the git side of development
       history in this repository starts at that point for me. 

     . "git branch origin"; the tip of CVS repository is kept
       track of with this branch.  I work in "master".

     I think I could have done the above with git-cvsimport,

 (1) Beginning of the day.  In case other people did work on
     the CVS side, I do this:

     . "git checkout origin", "cvs -q update".  If there is no
       change, go to step (3).

     . add any new files with "git add", and update the "origin"
       branch with "git commit -a -m 'from CVS'".

 (2) Merge other's work into my git master branch.

     . "git checkout master" and "git pull . cvs"; conflict
       resolve as needed.

 (3) Do my work.

     . "git checkout master" if I haven't done so.     

     . hack away, grow "master" branch using full power of git
       including the use of topic branches etc.

 (4) Publish, when "master" changes are ready.

     . To avoid conflicts with other people working on CVS,
       perform (1) again to make sure "origin" matches the tip
       of CVS.

     . "git checkout origin", "git pull . master".

     . generate the consolidated log I am about to push back to
       CVS with "git log --no-merges ORIG_HEAD.. | git shortlog >L".

     . add any new files with "cvs add", and "cvs commit -F L"

     . go back to (3) and continue.

This can be extended to multiple CVS branches.  Per one CVS
branch, use one git branch to track its head (plays the role of
"origin" in the above), use another to do the real development
for that branch ("master" in the above).  Never use merges on
the CVS side, but perform merges on the git side and then
pushing the merge result back to CVS.  This frees you from the
nightmarish branch management.

The above obviously have advantages and disadvantages.

 . I do not use any git import tool, so in theory you can apply
   this to _any_ foreign SCM.

 . I treat the foreign SCM as a second class citizen; multiple
   commits made there are rolled into a single commit on the git
   side (the real project history as far as I am concerned),
   which may or may not be nice.  The people who see my work
   through CVS see multiple commits on git side as a single
   commit in CVS.

In my situation this is fine as is, but I think you could leave
the management of "origin" to git-cvsimport (then your working
tree would not have CVS metadirectories as the above outline
does) if you wanted to avoid the "rolled into one commit" issue.

Pushing the changes on the git side back to CVS may become
trickier (and that is why I still keep the above setup without
using git-cvsimport), as I do not offhand know how exportcommit
works.  The commits on the git side you would want to push back
are the ones on "master" but not on "origin", which you can get
from "git rev-list origin..master", so I presume if you feed
that to exportcommit things all magically work?

By the way, I met Paul a few days ago and he mentioned that you
have some quick audit results on our code from your evaluation.
Can we expect fixes or at least pointing-out-problems from you
sometime soon please?

Re: synchronizing incremental git changes to cvs

[Jim Meyering]

Junio C Hamano <junkio@cox.net> wrote:
> Jim Meyering <jim@meyering.net> writes:
>> I'd like to develop using git, and have a commit hook mirror the
>> day-to-day changes (tags/commits) made in the git repo to a
>> cvs repository.  The idea is that the only way changes get into
>> the cvs repo is via the git commit hook.
>
> I do not use the automated tools myself, but I sync the day-job
> work in my git repository to CVS at work.  I do not develop with
> CVS but use it merely as a publishing medium.  Although other
> people can make commits into CVS in which case I have to slurp
> the change back into my git repository.
>
>  (0) Bootstrap.  I did use git-cvsimport myself (this repository
...
>  (1) Beginning of the day.  In case other people did work on
...

Thank you for describing the process you use.
However, since I don't have to allow independent cvs commits,
I hope to bend git-cvsexportcommit to my needs.
It already does almost everything I want.

> using git-cvsimport), as I do not offhand know how exportcommit
> works.  The commits on the git side you would want to push back
> are the ones on "master" but not on "origin", which you can get
> from "git rev-list origin..master", so I presume if you feed
> that to exportcommit things all magically work?

I haven't yet tried to restrict the mirroring to commits on a specific
git branch.  So far, in the toy example I'm using to test things,
I have this in .git/hooks/post-commit:

#!/bin/sh
sha1_id=$(git-rev-parse --verify HEAD)
cvsdir=/var/tmp/work-c
cd $cvsdir && GIT_DIR=/var/tmp/git-experiment/work-g/.git \
    git-cvsexportcommit -v -c -p $sha1_id

I'll clean up and post the changes I've made to git-cvsexportcommit
this week.

> By the way, I met Paul a few days ago and he mentioned that you
> have some quick audit results on our code from your evaluation.
> Can we expect fixes or at least pointing-out-problems from you
> sometime soon please?

I'll send one report separately.

don't accept bogus N in `HEAD~N'

[Jim Meyering]

In a very shallow audit, I spotted code where overflow was not detected.
But it's hardly critical.

Currently,

  git-diff HEAD HEAD

is equivalent to this

  git-diff HEAD HEAD~18446744073709551616   # aka 2^64

Exercising git-rev-parse directly, currently I get this:

  $ git-rev-parse --no-flags --sq HEAD~18446744073709551616
  '639ca5497279607665847f2e3a11064441a8f2a6'

It'd be better to produce a diagnostic and fail:

  $ ./git-rev-parse --no-flags --sq -- HEAD~18446744073709551616 > /dev/null
  fatal: ambiguous argument 'HEAD~18446744073709551616': unknown revision or filename

The code in question is in sha1_name.c (get_sha1_1):

               int num = 0;
               ...
               while (cp < name + len)
                       num = num * 10 + *cp++ - '0';

Looking at how to fix it, my first reflex was to replace that loop
with this one:

		while (cp < name + len) {
			int tmp = num * 10 + *cp++ - '0';
			if (INT_MAX / 10 < num || tmp < num)
				return -1;
			num = tmp;
		}

But INT_MAX is used nowhere else, so I wonder if git avoids using
it for some reason.  At least `make check' gripes about __INT_MAX__.
Anyhow, here's the patch I used.  With it, git still passes `make test'.

diff --git a/sha1_name.c b/sha1_name.c
index dc68355..c813ba0 100644
--- a/sha1_name.c
+++ b/sha1_name.c
@@ -429,8 +429,12 @@ static int get_sha1_1(const char *name,
 		int num = 0;
 		int len1 = cp - name;
 		cp++;
-		while (cp < name + len)
-			num = num * 10 + *cp++ - '0';
+		while (cp < name + len) {
+			int tmp = num * 10 + *cp++ - '0';
+			if (INT_MAX / 10 < num || tmp < num)
+				return -1;
+			num = tmp;
+		}
 		if (has_suffix == '^') {
 			if (!num && len1 == len - 1)
 				num = 1;

Re: don't accept bogus N in `HEAD~N'

[Jakub Narebski]

Jim Meyering wrote:

> In a very shallow audit, I spotted code where overflow was not detected.
> But it's hardly critical.
> 
> Currently,
> 
>   git-diff HEAD HEAD
> 
> is equivalent to this
> 
>   git-diff HEAD HEAD~18446744073709551616   # aka 2^64
> 
> Exercising git-rev-parse directly, currently I get this:
> 
>   $ git-rev-parse --no-flags --sq HEAD~18446744073709551616
>   '639ca5497279607665847f2e3a11064441a8f2a6'
> 
> It'd be better to produce a diagnostic and fail:
> 
>   $ ./git-rev-parse --no-flags --sq -- HEAD~18446744073709551616 /dev/null
>   fatal: ambiguous argument 'HEAD~18446744073709551616': unknown revision or filename

Wouldn't it remove ability to say "to the root commit"?
One can do it now I guess exactly by specyfying overly large N.
Although there should probably be some limit... or not.

-- 
Jakub Narebski
Warsaw, Poland

Re: don't accept bogus N in `HEAD~N'

[Jim Meyering]

Jakub Narebski <jnareb@gmail.com> wrote:

> Jim Meyering wrote:
>
>> In a very shallow audit, I spotted code where overflow was not detected.
>> But it's hardly critical.
>>
>> Currently,
>>
>>   git-diff HEAD HEAD
>>
>> is equivalent to this
>>
>>   git-diff HEAD HEAD~18446744073709551616   # aka 2^64
>>
>> Exercising git-rev-parse directly, currently I get this:
>>
>>   $ git-rev-parse --no-flags --sq HEAD~18446744073709551616
>>   '639ca5497279607665847f2e3a11064441a8f2a6'
>>
>> It'd be better to produce a diagnostic and fail:
>>
>>   $ ./git-rev-parse --no-flags --sq -- HEAD~18446744073709551616 /dev/null
>>   fatal: ambiguous argument 'HEAD~18446744073709551616': unknown revision or filename
>
> Wouldn't it remove ability to say "to the root commit"?
> One can do it now I guess exactly by specyfying overly large N.
> Although there should probably be some limit... or not.

Do people really use HEAD~<VERY_LARGE_INTEGER> to refer to the root?
Any who do that will find it surprising that HEAD~18446744073709551616
is currently interpreted just like `HEAD~0'.
And HEAD~18446744073709551617 just like HEAD~1, etc.

Re: don't accept bogus N in `HEAD~N'

[Junio C Hamano]

Jim Meyering <jim@meyering.net> writes:

> Jakub Narebski <jnareb@gmail.com> wrote:
>
>> Jim Meyering wrote:
>>
>>> It'd be better to produce a diagnostic and fail:

I agree with you that we are loose in integer overlaps.  Some of
them do matter, some don't.  The xrealloc one is, as you said,
borderline, I think, but more serious than this one.  This one
is worth fixing only if/because the fix is obvious and does not
hurt the code otherwise (e.g. does not decrease portability,
does not hurt usability, etc.).

>>>
>>>   $ ./git-rev-parse --no-flags --sq -- HEAD~18446744073709551616 /dev/null
>>>   fatal: ambiguous argument 'HEAD~18446744073709551616': unknown revision or filename
>>
>> Wouldn't it remove ability to say "to the root commit"?
>> One can do it now I guess exactly by specyfying overly large N.
>> Although there should probably be some limit... or not.
>
> Do people really use HEAD~<VERY_LARGE_INTEGER> to refer to the root?

You shouldn't have to care about nor refer to the root commit
that often (if ever) in a real project.  It is handy to be able
to refer to it when your repository is very young and you are
toying with git more than you are working on your own project
that is managed by git.  But in such a case, finding it once and
tagging it is so easy and efficient that you would not want to
traverse the whole history every time you would want to refer to
it.

In other words, I think Jakub was just joking, and this
particular objection does not qualify as "hurt usability"
criteria I said in the above.

Re: don't accept bogus N in `HEAD~N'

[Junio C Hamano]

Junio C Hamano <junkio@cox.net> writes:

> Jim Meyering <jim@meyering.net> writes:
>
>> Jakub Narebski <jnareb@gmail.com> wrote:
>>
>>> Jim Meyering wrote:
>>>
>>>> It'd be better to produce a diagnostic and fail:
>
> I agree with you that we are loose in integer overlaps.  Some of

Oops; I meant overflow or wraparound.. Late night typo/thinko X-<. 

avoid atoi, when possible; int overflow -> heap corruption

[Jim Meyering]

This is another one of those `would be nice' sort of changes.
Probably not worth much at this early stage in development, but
eventually worth changing.

There are about 20 uses of atoi, and most calls can return
a usable result in spite of an invalid input -- just because
atoi returns the same thing for "99" as "99-and-any-suffix".
It would be better not to ignore invalid inputs.

-------------------
Also, integer overflow in object.c can cause trouble.
When the xrealloc byte count exceeds 2^32 (for a 32-bit int),
xrealloc will happily return a buffer of the requested (small) size,
but the following memset will scribble zeroes far beyond the end
of that new buffer.

static int nr_objs;
int obj_allocs;
...
void created_object(const unsigned char *sha1, struct object *obj)
{
...
	if (obj_allocs - 1 <= nr_objs * 2) {
		int i, count = obj_allocs;
		obj_allocs = (obj_allocs < 32 ? 32 : 2 * obj_allocs);
		objs = xrealloc(objs, obj_allocs * sizeof(struct object *));
		memset(objs + count, 0, (obj_allocs - count)
				* sizeof(struct object *));

But this may be only theoretical, because the problem doesn't strike
until there are over 250M objects (assuming 32-bit int and 8-byte pointers).

Re: avoid atoi, when possible; int overflow -> heap corruption

[Morten Welinder]

> There are about 20 uses of atoi, and most calls can return
> a usable result in spite of an invalid input -- just because
> atoi returns the same thing for "99" as "99-and-any-suffix".
> It would be better not to ignore invalid inputs.

atoi has undefined behaviour for "99-and-any-suffix".  You might
get lucky and get back 99, but you might also get a random value
or a core dump.

Morten

Re: avoid atoi, when possible; int overflow -> heap corruption

[Jim Meyering]

"Morten Welinder" <mwelinder@gmail.com> wrote:
>> There are about 20 uses of atoi, and most calls can return
>> a usable result in spite of an invalid input -- just because
>> atoi returns the same thing for "99" as "99-and-any-suffix".
>> It would be better not to ignore invalid inputs.
>
> atoi has undefined behaviour for "99-and-any-suffix".  You might
> get lucky and get back 99, but you might also get a random value
> or a core dump.

I've never heard of that.
POSIX says that atoi(str) is equivalent to:

    (int) strtol(str, (char **)NULL, 10)
    except that the handling of errors may differ.
    If the value cannot be represented, the behavior is undefined.

Since strtol works fine with such a suffix, and since 99 can be
represented, I don't see why there would be any undefined behavior.

Do you know of an implementation for which `atoi ("99-and-any-suffix")'
does anything other than return 99?

Re: avoid atoi, when possible; int overflow -> heap corruption

[Jeff King]

On Mon, May 22, 2006 at 09:16:50AM -0400, Morten Welinder wrote:

> atoi has undefined behaviour for "99-and-any-suffix".  You might
> get lucky and get back 99, but you might also get a random value
> or a core dump.

Where do you get that from? The standard claims that it converts "the
initial portion of the string pointed to" (7.20.1.2). Furthermore, atoi
is equivalent to strtol with a base of 10 (with the exception of range
errors). From 7.20.1.4, paragraph 2:
  The strtol [...] functions [...] decompose the input string into three
  parts: an initial, possibly empty, sequence of white-space characters
  [...], a subject sequence resembling an integer represented in some
  radix determined by the value of base, and a final string of one or
  more unrecognized characters...
If no conversion can be performed (i.e., you feed it garbage with no
number), zero is returned.

atoi does NOT handle range errors, however; the behavior is undefined in
that case. In practice, I expect most implementations do some sort of
wrapping.

-Peff

Re: avoid atoi, when possible; int overflow -> heap corruption

[Morten Welinder]

My copy (which is admittedly a draft because I am cheap) does not
restrict undefined behaviour to _range_ errors, but simply says
"Except for the behavior on error, they are equivalent to [the strtol call]"

M.

detect write failure, even for stdout

[Jim Meyering]

git doesn't always detect write failures.  A write I/O error,
(e.g., hardware I/O error or simply disk full)
doesn't provoke nonzero exit status:

    $ ./git-cat-file -t HEAD > /dev/full && echo did not detect write failure
    did not detect write failure

This is perhaps more important than the other things
I've reported, since it can lead to porcelain being unable
to detect a real failure in the plumbing.

Here are two more:

    $ ./git-ls-tree HEAD > /dev/full && echo fail
    fail
    $ ./git-show > /dev/full && echo fail
    fail

If you were using gnulib, I'd suggest simply adding this line

    atexit (close_stdout);

near the beginning of each `main'.  Then you wouldn't have to
manually track down each and every place where a write to stdout
can occur -- not to mention the maintenance burden of keeping
things correct as the code evolves.