remaining git-cvsimport problems: robustness when cvsps feeds strange history

remaining git-cvsimport problems: robustness when cvsps feeds strange history

[Yann Dirson]

Since there is quite some activity around git-cvsimport these days, I
think I'd raise a couple of issues which are still problematic.

It should at least issue a warning when cvsps feeds it with a
branchpoint that it cannot create because the base branch has not yet
been seen.  I am still investigating why cvsps outputs such a buggy
patchset - it may be a cvs bug at the root, and cvsps probably did not
detect an consistency of the cvs data.  While there is probably a real
bug to fix somewhere, git-cvsimport should not silently skip the
faulty patchset, especially as it will then miserably fail to import
any subsequent patch on that branch it ignored this way.

However, rerunning the script (after creating a master head by copying
heads/origin) manages to complete the import: the forgotten branch now
gets created, and all subsequent patchsets can now be imported.


I'm not sure what would be the best way to handle the issue.  Maybe it
would sufficient to output a bold warning and let the user take the
necessary steps to finish the import (since this is after all a bug in
other tools, which will hopefully be fixed at some point).  It could
also be useful to raise a flag when detecting that condition, and
automatically restart the import process on the first error due to
that branch not being available - otoh, that may be overkill, and
could possibly miss some other obscure cases.

As a sidenote, I'm wondering why there is no precise information on
the branchpoint in "cvsps -A".  I guess the semantics are "fork a new
branch from the ancestor one" at whatever point it currently is - that
would look quite risky to me, and could be part of the reason why
cvsps did not notice the inconsistency: it just did not try to find
out where the new branch was to be grafted exactly.

======
PatchSet 1
Date: 2006/01/03 18:23:53
Author: ydirson
Branch: HEAD
Tag: (none)
Log:
Empty .cvsignore to be used as a branching base

Members:
        .cvsignore:INITIAL->1.1

---------------------
PatchSet 2
Date: 2006/01/03 18:23:53
Author: ydirson
Branch: FOO_V3_0_0_HEAD
Ancestor branch: FOO_VENDOR_HEAD
Tag: (none)
Log:
file .cvsignore was added on branch FOO_V3_0_0_HEAD on 2006-05-24 07:51:07
+0000

Members:
        .cvsignore:1.1->1.1.4.1(DEAD)

---------------------
PatchSet 3
Date: 2006/01/03 18:24:39
Author: ydirson
Branch: FOO_VENDOR_HEAD
Ancestor branch: HEAD
Tag: FOO_V1_1
Log:
Import de FOO 1.1
---------------------
PatchSet 20
Date: 2006/01/10 11:47:54
Author: ydirson
Branch: FOO_V3_0_0_HEAD
Tag: (none)
Log:
file script.sh was added on branch FOO_V3_0_0_HEAD on 2006-05-24 07:51:08
+0000

Members:
        some/script.sh:1.1->1.1.4.1(DEAD)
===== first import
Fetching .cvsignore   v 1.1
New .cvsignore: 0 bytes
Tree ID 87556a35bc2c438ede9eb2120c2cdb04baed33ae
Committing initial tree 87556a35bc2c438ede9eb2120c2cdb04baed33ae
Committed patch 1 (origin 2006-01-03 17:23:53)
Commit ID 2ddb63602915316e60b26d95414100fd80e602ef
Branch FOO_VENDOR_HEAD does not exist!
Delete .cvsignore
[...]
Tree ID ca6e986b5199b01604dfa459b6e3a2841024f11a
Parent ID 2ddb63602915316e60b26d95414100fd80e602ef
Committed patch 3 (FOO_VENDOR_HEAD 2006-01-03 17:24:39)
Commit ID 9e19f11520c89897704fff252901ff4dc5ae88ad
[...]
Committed patch 19 (origin 2006-01-10 10:47:54)
Commit ID 0c5063420b769f8db7c009843b3947350bf78eae
Switching from origin to FOO_V3_0_0_HEAD
fatal: Not a valid object name FOO_V3_0_0_HEAD
read-tree failed: 32768
===== second run
skip patchset 1: 1136309033 before 1136890074
Switching from master to FOO_V3_0_0_HEAD
Delete .cvsignore
Tree ID 38b8067e1d36ce7f45bb1121a22628927bfd2ac2
Parent ID b9cf667acd27ba7fd76a405166b355ec51261b17
Committed patch 2 (FOO_V3_0_0_HEAD 2006-01-03 17:23:53)
Commit ID 528efdac6113af50d4c14c3ccabde9661cb19359
skip patchset 3: 1136309079 before 1136312985
[...]
skip patchset 19: 1136890074 before 1136890074
Delete some/script.sh
Tree ID 38b8067e1d36ce7f45bb1121a22628927bfd2ac2
Parent ID 528efdac6113af50d4c14c3ccabde9661cb19359
Committed patch 20 (FOO_V3_0_0_HEAD 2006-01-10 10:47:54)
[...]
Committed patch 409 (origin 2006-05-24 18:59:44)
Commit ID 74a5c3b1f265f096177120166ae7da1a2cf3f64e
DONE.
=====
--
Yann Dirson    <ydirson@altern.org> |
Debian-related: <dirson@debian.org> |   Support Debian GNU/Linux:
                                    |  Freedom, Power, Stability, Gratis
     http://ydirson.free.fr/        | Check <http://www.debian.org/>

Re: remaining git-cvsimport problems: robustness when cvsps feeds strange history

[Martin Langhoff]

Yann,

I want to see if we can close these gaps. Have you got a public repo
that shows this problem so can look more into it?

On 5/28/06, Yann Dirson <ydirson@altern.org> wrote:
> As a sidenote, I'm wondering why there is no precise information on
> the branchpoint in "cvsps -A".  I guess the semantics are "fork a new
> branch from the ancestor one" at whatever point it currently is - that
> would look quite risky to me, and could be part of the reason why
> cvsps did not notice the inconsistency: it just did not try to find
> out where the new branch was to be grafted exactly.

It is perfectly possible for cvs to branch at a "point" that is not
really a patchset/patchlevel. Just like it is to tag something that
has never been a patchset.

It is something we currently fudge a bit (or a lot, depending on your
point of view). If the branch was made on a checkout with an
inconsistent tree, we cannot really represent that in git matching
what happened in CVS.

OTOH, the cvsps output you are showing us seems to be in the right
order...  patchset 20 should go on top of patchset 3... is cvsimport
truly mishandling this?



martin

Re: remaining git-cvsimport problems: robustness when cvsps feeds strange history

[Yann Dirson]

On Sun, May 28, 2006 at 03:23:01AM +1200, Martin Langhoff wrote:
> I want to see if we can close these gaps. Have you got a public repo
> that shows this problem so can look more into it?

No, it holds proprietary code - I can only try to reproduce the
history pattern, but I have not succeeded in that yet.  And that's not
something I can do on customer time :)


> On 5/28/06, Yann Dirson <ydirson@altern.org> wrote:
> >As a sidenote, I'm wondering why there is no precise information on
> >the branchpoint in "cvsps -A".  I guess the semantics are "fork a new
> >branch from the ancestor one" at whatever point it currently is - that
> >would look quite risky to me, and could be part of the reason why
> >cvsps did not notice the inconsistency: it just did not try to find
> >out where the new branch was to be grafted exactly.
> 
> It is perfectly possible for cvs to branch at a "point" that is not
> really a patchset/patchlevel. Just like it is to tag something that
> has never been a patchset.
>
> It is something we currently fudge a bit (or a lot, depending on your
> point of view). If the branch was made on a checkout with an
> inconsistent tree, we cannot really represent that in git matching
> what happened in CVS.

Yes, I know that too much.  I've already thought that it could
possibly be represented by a merge commit taking as parents all
patchsets that contain a file revision holding the tag.  But such
patchset synthesis ought to done at cvsps level IMHO.


> OTOH, the cvsps output you are showing us seems to be in the right
> order...  patchset 20 should go on top of patchset 3... is cvsimport
> truly mishandling this?

That's the problem.  I had just copypasted the logs for handling at
home, and then I discovered in the cvsps log that the timestamp for
patchset 2 is the same as the one for patchset 1, which is obviously
wrong.  I'll look at the cvs repo next week to understand whether that
comes from the RCS file, or whether it is cvsps misunderstanding
something.

Best regards,
-- 
Yann Dirson    <ydirson@altern.org> |
Debian-related: <dirson@debian.org> |   Support Debian GNU/Linux:
                                    |  Freedom, Power, Stability, Gratis
     http://ydirson.free.fr/        | Check <http://www.debian.org/>

Re: remaining git-cvsimport problems: robustness when cvsps feeds strange history

[Yann Dirson]

On Sat, May 27, 2006 at 06:35:55PM +0200, Yann Dirson wrote:
> > OTOH, the cvsps output you are showing us seems to be in the right
> > order...  patchset 20 should go on top of patchset 3... is cvsimport
> > truly mishandling this?
> 
> That's the problem.  I had just copypasted the logs for handling at
> home, and then I discovered in the cvsps log that the timestamp for
> patchset 2 is the same as the one for patchset 1, which is obviously
> wrong.  I'll look at the cvs repo next week to understand whether that
> comes from the RCS file, or whether it is cvsps misunderstanding
> something.

The problem indeed stems from the RCS files themselves.  It thus comes
from a bug-or-feature of cvs 1.12.12.  This week I have found a number
of files with such an history in various repos in use at work.

I finally realize I was trying to reproduce the problem at home using
1.12.9, where the issue does not appear to exist.  The following
script does show the problem with 1.12.13.

As you see, a phantom commit is added prior to the resurecting commit
on the branch, and it has the same timestamp as the its dead base
revision on the trunk.

In the logs I had from real repositories with 1.12.12, the "lines"
attribute for the phantom revision was a mirror of the revision
following it, reflecting no reality at all (would have been "+0 -1"
here).  There is no mention of such a bugfix in the cvs NEWS file, but
who knows.

I found no mention of such a feature in the 1.12.9-13 entries in the
CVS NEWS file, so I doubt that is an intended behaviour - I'll report
a bug on cvs.

However, cvsps should surely be made aware of this behaviour, even if
it is a bug, since being there for at least 2 cvs releases is enough
to have infected many repositories...


==== script
#!/bin/sh
set -e

# setup repo and working area
cvs -d $PWD/root init
mkdir root/test
cvs -d $PWD/root co test
cd test

# trunk
touch file1 file2
cvs add file1 file2
cvs ci -m "add files"
sleep 2

rm file1 
cvs rm file1 
cvs ci -m "rm file1"
sleep 2

cvs tag A

# branch based on trunk
cvs tag -b A_HEAD
cvs up -r A_HEAD -P
echo "stuff and more stuff" >file1
cvs add file1
cvs ci -m "work"
sleep 2

cvsps --norc -x -A
==== cvs log file1
----------------------------
revision 1.2
date: 2006-06-01 23:12:21 +0200;  author: dwitch;  state: dead;  lines: +0 -0;  commitid: GyCIctKzKqKVwlzr;
branches:  1.2.2;
rm file1
----------------------------
revision 1.1
date: 2006-06-01 23:12:17 +0200;  author: dwitch;  state: Exp;  commitid: j1zLyPu2Bw6Uwlzr;
add files
----------------------------
revision 1.2.2.2
date: 2006-06-01 23:12:23 +0200;  author: dwitch;  state: Exp;  lines: +1 -0;  commitid: D3Fd6f9Wz8cWwlzr;
work
----------------------------
revision 1.2.2.1
date: 2006-06-01 23:12:21 +0200;  author: dwitch;  state: dead;  lines: +0 -0;  commitid: D3Fd6f9Wz8cWwlzr;
file file1 was added on branch A_HEAD on 2006-06-01 21:12:23 +0000
====

-- 
Yann Dirson    <ydirson@altern.org> |
Debian-related: <dirson@debian.org> |   Support Debian GNU/Linux:
                                    |  Freedom, Power, Stability, Gratis
     http://ydirson.free.fr/        | Check <http://www.debian.org/>