cvsps: bad usage: invalid argument --norc

cvsps: bad usage: invalid argument --norc

[Ilya Basin]

Hi esr.
In cvsps 3.10 the flag --norc was removed. It broke 'git cvsimport'.
Please give the option back and write something in the man page like:
    This option has no effect; it is present for compatibility

Re: cvsps: bad usage: invalid argument --norc

[Ilya Basin]

IB> Hi esr.
IB> In cvsps 3.10 the flag --norc was removed. It broke 'git cvsimport'.
IB> Please give the option back and write something in the man page like:
IB>     This option has no effect; it is present for compatibility

Looks like the tool is completely different. I think I'll have to
downgrade.

-- 

Re: cvsps: bad usage: invalid argument --norc

[John Keeping]

On Sun, Apr 14, 2013 at 12:48:00PM +0400, Ilya Basin wrote:
> IB> Hi esr.
> IB> In cvsps 3.10 the flag --norc was removed. It broke 'git cvsimport'.
> IB> Please give the option back and write something in the man page like:
> IB>     This option has no effect; it is present for compatibility
> 
> Looks like the tool is completely different. I think I'll have to
> downgrade.

If you want incremental import support, then that's the best thing to
do.  There was some discussion about this in January [1] and it doesn't
look like cvsps-3 has changed since then, so if you want to
incrementally update a Git clone of a CVS repository then cvsps-2 is
really the only way to go.

If you're doing a one-off import then I recommend ignoring git-cvsimport
and investigating cvs2svn[2] and cvs-fast-export [3].

[1] http://thread.gmane.org/gmane.comp.version-control.git/214258/focus=214305
[2] http://cvs2svn.tigris.org/cvs2git.html
[3] https://gitorious.org/cvs-fast-export

State of CVS-to-git conversion tools (Was: Re: cvsps: bad usage: invalid argument --norc)

[Eric S. Raymond]

Ilya Basin <basinilya@gmail.com>:
> IB> Hi esr.
> IB> In cvsps 3.10 the flag --norc was removed. It broke 'git cvsimport'.
> IB> Please give the option back and write something in the man page like:
> IB>     This option has no effect; it is present for compatibility
> 
> Looks like the tool is completely different. I think I'll have to
> downgrade.

Or you could just use 3.x directly rather than through the git-cvsimport
wrapper.  It works better that way - it actually ships a fast-import 
stream that git fast-import can consume directly.

Old cvsps (2.x) was very, very broken; there was a bug in it that
pretty much guaranteed incorrect conversion of branchy repos.  I've
fixed that particular bug, and several other serious ones, along with
adding the import-stream output stage, but I don't really trust the
cvsps code; I think its algorithmic core is weak and klugey and only
works semi-by-accident.

I do *not* recommend downgrading, it will pitch you from a bad
situation into a worse one.  Yes, Junio is still shipping a wrapper
for 2.x, but that was very much against my advice.

I'm also now the maintainer of cvs-fast-export, which used to be Keith 
Packard's utility for lifting the X.org CVS repo to git.  That was before
I resurrected it and added a fast-import stream output stage. 

I *think* cvs-fast-export's algorithms are more correct than cvsps's,
but I have not yet been able to pry loose the time to write the really
rigorous test suite to verify this.  That goal has been second on my
prority list for a couple of months now; I keep wanting to get to it
but having to fight fires on other projects instead.

I wish I could point you at a convertor I really trust.  I can't.
There is a third tool, cvs2git (based on the analyzer from cvs2svn)
that I don't maintain, which has problems of its own. And those three
are about it.

Yes, it's a swamp. The relatively poor capability of the tools isn't
anybody's fault; the problem domain is *nasty*.  I've been working a
closely related but easier one (Subversion stream dump analysis) for a
couple years now and understanding it doesn't make it less ugly.

I think with 4-6 weeks of concentrated attention I could clean up the
mess and deliver a really high-quality converter, and I'm motivated to
do this because I want it as a CVS front end for reposurgeon.  But it
hasn't happened yet and the incompleteness of my test suite is a
blocker in the way.

The topo analysis code in all these tools is really fragile and tends
to break on old edge cases when you try to teach it to handle new
ones, so a good set of regression tests is especially important.  And
doesn't yet exist, though I have built a decent start for cvsps based
on the tests in the git tree.

Do you have enough interest and spare cycles to help finish the test
suite?  Another pair of hands on it might speed things up a lot.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

Re: State of CVS-to-git conversion tools (Was: Re: cvsps: bad usage: invalid argument --norc)

[Michael Haggerty]

On 04/14/2013 01:33 PM, Eric S. Raymond wrote:
> [...]
> I wish I could point you at a convertor I really trust.  I can't.
> There is a third tool, cvs2git (based on the analyzer from cvs2svn)
> that I don't maintain, which has problems of its own.
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Again I request that instead of spreading non-specific FUD about
cvs2git, you describe concrete problems so that I can improve the tool.
 So far about the only problems that you have mentioned that have not
been addressed have been stylistic trivia like that you object that
cvs2git writes its outputs by pass, to two separate files, rather than
writing all of its output to stdout, and that you are bothered that it
names its temporary directory "cvs2svn-tmp" rather than "cvs2git-tmp".

I would like to fix any conversion deficiencies you can find.  I stand
behind this tool and think it is the best one out there for extracting
history from CVS.

Michael

-- 
Michael Haggerty
mhagger@alum.mit.edu
http://softwareswirl.blogspot.com/

Re: State of CVS-to-git conversion tools (Was: Re: cvsps: bad usage: invalid argument --norc)

[Ilya Basin]

[-- Attachment #1: Type: text/plain, Size: 527 bytes --]

Hi Eric.

I tried --fast-export. It's 2 times faster.
The first thing that differs: in cvsps2 commits with adjacent
timestamps were joined into one (see the attached files). Do you know
the reason?

Does this --fast-export thing support what John mentioned, the
"incremental import support"? Does 'git fast-import' has it?
I need it, because full import takes too long.
The central repo of my employer is CVS, other people commit to it and
I use git internally to be able to tidy my commit history before
exporting to CVS.

-- 

[-- Attachment #2: cvsps2.txt --]
[-- Type: TEXT/PLAIN, Size: 1653 bytes --]

cvsps2:

commit c0654845791da1eefb1e223c8001169d61f9de51
Author: reevzh1 <reevzh1>
Date:   Fri Jul 28 10:47:17 2006 +0000

    no message

 components/rapportlets/src/com/sicap/guif/portalframework/portlets/rap/ReportFraudCDRQuerierPortlet.java | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

commit 2ed1c92eec518a5752277a16cf2aea6bd488d6be
Author: reevzh1 <reevzh1>
Date:   Fri Jul 28 04:51:06 2006 +0000

    no message

 components/rapportlets/src/com/sicap/guif/portalframework/portlets/rap/CommonAttributeNames.java         |   6 ++
 components/rapportlets/src/com/sicap/guif/portalframework/portlets/rap/ReportFraudCDRQuerierItem.java    | 129 ++++++++++++++++++++++++
 components/rapportlets/src/com/sicap/guif/portalframework/portlets/rap/ReportFraudCDRQuerierPortlet.java | 101 +++++++++++++++----
 components/rapportlets/src/com/sicap/guif/portalframework/portlets/rap/containers/ReportContainer.java   |  48 +++++++++
 resources/ReportFraudCDRQuerier_en.properties                                                            |   2 +
 web/jsp/report_fraudcdr_querier.jsp                                                                      |  21 ++--
 web/jsp/report_fraudcdr_querier_print.jsp                                                                | 309 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 web/resources/Language_en.properties                                                                     |   8 +-
 8 files changed, 595 insertions(+), 29 deletions(-)









commit d6a7335ea24323509df02b08f52af01926eb5e4d
Author: redane1 <redane1>
Date:   Thu Jul 6 10:28:24 2006 +0000

    no message

...

[-- Attachment #3: cvsps3.txt --]
[-- Type: TEXT/PLAIN, Size: 1872 bytes --]

cvsps3:

commit d3732dbdf41b348f0050e888733ce2afec05f7f8
Author: reevzh1 <reevzh1>
Date:   Fri Jul 28 10:47:17 2006 +0000

    no message

 components/rapportlets/src/com/sicap/guif/portalframework/portlets/rap/ReportFraudCDRQuerierPortlet.java | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

commit 158bc55affa80b60471ea6026b1393934f928e58
Author: reevzh1 <reevzh1>
Date:   Fri Jul 28 04:51:22 2006 +0000

    no message

 components/rapportlets/src/com/sicap/guif/portalframework/portlets/rap/ReportFraudCDRQuerierItem.java  | 129 +++++++++++++++++++++++++
 components/rapportlets/src/com/sicap/guif/portalframework/portlets/rap/containers/ReportContainer.java |  48 ++++++++++
 web/jsp/report_fraudcdr_querier_print.jsp                                                              | 309 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 web/resources/Language_en.properties                                                                   |   8 +-
 4 files changed, 493 insertions(+), 1 deletion(-)

commit 1ba90a2baebee9f1838dab04f129c7704331e0a8
Author: reevzh1 <reevzh1>
Date:   Fri Jul 28 04:51:06 2006 +0000

    no message

 components/rapportlets/src/com/sicap/guif/portalframework/portlets/rap/CommonAttributeNames.java         |   6 ++++
 components/rapportlets/src/com/sicap/guif/portalframework/portlets/rap/ReportFraudCDRQuerierPortlet.java | 101 +++++++++++++++++++++++++++++++++++++++++++++------------
 resources/ReportFraudCDRQuerier_en.properties                                                            |   2 ++
 web/jsp/report_fraudcdr_querier.jsp                                                                      |  21 ++++++++----
 4 files changed, 102 insertions(+), 28 deletions(-)

commit d6a7335ea24323509df02b08f52af01926eb5e4d
Author: redane1 <redane1>
Date:   Thu Jul 6 10:28:24 2006 +0000

    no message

...

Re: State of CVS-to-git conversion tools (Was: Re: cvsps: bad usage: invalid argument --norc)

[Jakub Narębski]

Ilya Basin wrote:

> Does this --fast-export thing support what John mentioned, the
> "incremental import support"? Does 'git fast-import' has it?
> I need it, because full import takes too long.
>
> The central repo of my employer is CVS, other people commit to it and
> I use git internally to be able to tidy my commit history before
> exporting to CVS.

Errr... the need to interact with *live* CVS repository is reason 
behind wanting incremental import, isn't it?

I wonder how hard would be to create a remote helper for CVS, so that 
one could treat CVS repository as foreign remote, fetching from it and 
pushing to it...

BTW. can you convince your employer to move to more modern version 
control system (be it Subversion, Kiln Harmony, Veracity, Bazaar, 
Mercurial or Git) rather than rely on system which had last release in 
2008 i.e. 4 years ago, doesn't support atomic commits, doesn't support 
versioning of whole repository aka. changesets (beside tags), has slow 
and complicated branching and even more complicated merging, etc.?

http://stackoverflow.com/questions/802573/difference-between-git-and-cvs/824241#824241
-- 
Jakub Narębski

Re: State of CVS-to-git conversion tools (Was: Re: cvsps: bad usage: invalid argument --norc)

[Eric S. Raymond]

Ilya Basin <basinilya@gmail.com>:
> Hi Eric.
> 
> I tried --fast-export. It's 2 times faster.
> The first thing that differs: in cvsps2 commits with adjacent
> timestamps were joined into one (see the attached files). Do you know
> the reason?

The cvsps guy included code to do that. Keith Packard didn't.  
Sorry I can't be more helpful, but that's about all I know.

I didn't write either analysis stage; I understand cvsps's, somewhat,
because I had to fix several nasty bugs in it.  I *don't* understand
cvs-fast-export's analysis stage very well yet, because it has no
obvious bugs that have required me to dive in.  (Keith's notes
document one major bug, which may be inherent to the mismatch between
file- and changest-orientation and not fixable in the general case,
though I will try.)

> Does this --fast-export thing support what John mentioned, the
> "incremental import support"? Does 'git fast-import' has it?

cvs-fast-export does not have incremental-import support.  Whether
git-cvs-import has it depend on which version you have and what
backend it it is using. I don't maintain that wrapper.

> I need it, because full import takes too long.
> The central repo of my employer is CVS, other people commit to it and
> I use git internally to be able to tidy my commit history before
> exporting to CVS.

You are out of luck. That feature was dependent on a very fragile
coupling between the old output format and a bunch of unmaintainably 
horrible Perl in the git-cvs-import wrapper script.  It didn't
work very well; frankly, I'm amazed it worked at all.

The things I had to do to fix the serious bugs in cvsps2 and make it
output a fast-import stream had the side effect of breaking that
coupling. cvsps3 won't give you that feature. Dropping back to cvsps2
to keep that feature will expose you to the cvsps2 bugs.

I'm sorry these tools are such a mess.  I'm trying to fix that, but
it's hard, slow work.  The problems are deeply ugly and the edge cases
have poisoned spikes.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

Does git fast-import support deltas?

[Ilya Basin]

Hi list.
Here's what I mean:
1) a created a git repo from a foreign source using git fast-import
2) new commits were added to the foreign source

Can I create a fast-import input stream not containing the commits
already existing in my git repo and import it?

I tried to create such streams with:
    cvsps --fast-export -d ...
and from a shallow git repo, but the new commits are not imported
(unless the import stream contains a new branch)

Re: Does git fast-import support deltas?

[Felipe Contreras]

On Fri, Apr 19, 2013 at 12:29 AM, Ilya Basin <basinilya@gmail.com> wrote:
> Hi list.
> Here's what I mean:
> 1) a created a git repo from a foreign source using git fast-import
> 2) new commits were added to the foreign source
>
> Can I create a fast-import input stream not containing the commits
> already existing in my git repo and import it?

Yes, see the --import-marks and --export-marks options. The stream has
to refer to the marks that were used int the previous run.

Cheers.

-- 
Felipe Contreras

Re: Does git fast-import support deltas?

[Jonathan Nieder]

Hi Ilya,

Ilya Basin wrote:

> 1) a created a git repo from a foreign source using git fast-import
> 2) new commits were added to the foreign source
>
> Can I create a fast-import input stream not containing the commits
> already existing in my git repo and import it?

Yes, if the foreign source is structured in a way to make it easy.
Take a look at the --cat-blob-fd option, the "ls" and "cat-blob"
commands, and see svn-fe from contrib/svn-fe/ as an example.

Hope that helps,
Jonathan

Re[2]: State of CVS-to-git conversion tools (Was: Re: cvsps: bad usage: invalid argument --norc)

[Ilya Basin]

Hi Eric.

ESR> cvs-fast-export does not have incremental-import support.
ESR> Whether git-cvs-import has it depend on which version you have
ESR> and what backend it it is using. I don't maintain that wrapper.
Did you mean "git-fast-import"? Or do you know any wrapper that
already uses cvsps3 --fast-export?


>> I need it, because full import takes too long.
>> The central repo of my employer is CVS, other people commit to it and
>> I use git internally to be able to tidy my commit history before
>> exporting to CVS.

ESR> You are out of luck. That feature was dependent on a very fragile
ESR> coupling...
OK, OK, I get it.


First of all, I think cvsps3 has almost everithing required for
incremental import: one could just take the date of the last commit
and invoke cvs ps with the '-d' flag. However, to import new commits
into existing branches the stream should contain the "from" command in
oldest commits in each branch (now missing).
If the branch already exists in the target git repo, it's easy to
refer it in the stream:
    from refs/heads/branchname^0

But if the branch is new, but it's parent commit is already imported,
I guess, the only way to refer it is by its SHA-1
Eric, what parent information can cvsps provide for the first commit
in a branch, when invoked with the '-d' flag?

Re: State of CVS-to-git conversion tools (Was: Re: cvsps: bad usage: invalid argument --norc)

[Eric S. Raymond]

Apologies for the somewhat belated reply.  I've been even busier than
usual lately and am about to be traveling for a week.

Ilya Basin <basinilya@gmail.com>:
> Hi Eric.
> 
> ESR> cvs-fast-export does not have incremental-import support.
> ESR> Whether git-cvs-import has it depend on which version you have
> ESR> and what backend it it is using. I don't maintain that wrapper.
> Did you mean "git-fast-import"? Or do you know any wrapper that
> already uses cvsps3 --fast-export?

No, I meant git-cvs-import.  I wrote a version of it that supports
cvsps3, but Junio chose to keep the old wrapper.  Apparently he would
rather inflict cvsps2's rather serious known bugs on users than break
backward compatibility even a little.  

> First of all, I think cvsps3 has almost everithing required for
> incremental import: one could just take the date of the last commit
> and invoke cvs ps with the '-d' flag. However, to import new commits
> into existing branches the stream should contain the "from" command in
> oldest commits in each branch (now missing).
> If the branch already exists in the target git repo, it's easy to
> refer it in the stream:
>     from refs/heads/branchname^0

Look at the -i option.  That may do what you need.
 
> But if the branch is new, but it's parent commit is already imported,
> I guess, the only way to refer it is by its SHA-1
> Eric, what parent information can cvsps provide for the first commit
> in a branch, when invoked with the '-d' flag?

At the moment it doesn't provide any at all.  That case wasn't on my
radar when I was fixing the code.  If you can specify a behavior you
think would be useful, I'm listening.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

Re[2]: State of CVS-to-git conversion tools (Was: Re: cvsps: bad usage: invalid argument --norc)

[Ilya Basin]

>> But if the branch is new, but it's parent commit is already imported,
>> I guess, the only way to refer it is by its SHA-1
>> Eric, what parent information can cvsps provide for the first commit
>> in a branch, when invoked with the '-d' flag?

ESR> At the moment it doesn't provide any at all.  That case wasn't on my
ESR> radar when I was fixing the code.  If you can specify a behavior you
ESR> think would be useful, I'm listening.

For new branches the 'from' command can refer the common ancestor in
an existing branch. For example:

         /----------E thebranch
        /
    A---B---C---D master

Commit E is newer than D; we already imported D; thebranch is new.
Instead of:
    from refs/heads/thebranch^0
refer the parent as:
    from refs/heads/master^2



ESR> Look at the -i option.  That may do what you need.
ESR>

OK, something's wrong with the man page: starting with '-A' the
description is unstructured:

       -A authormap
           Apply an author-map file to the attribution lines. Each line must be of the form

               ferd = Ferd J. Foonly <foonly@foo.com> America/Chicago

           and will be applied to map the Unix username ferd to the DVCS-style user identity specified after the equals
           sign. The timezone field (after > and whitespace) is optional and (if present) is used to set the timezone
           offset to be attached to the date; acceptable formats for the timezone field are anything that can be in the TZ
           environment variable, including a [+-]hhmm offset. Whitespace around the equals sign is stripped. Lines
           beginning with a # or not containing an equals sign are silently ignored. -R revmap:: Write a revision map to
           the specified argument filename. Each line of the revision map consists of three whitespace-separated fields: a
           filename, an RCS revision number, and the mark of the commit to which that filename-revision pair was assigned.
           -v:: show very verbose parsing messages. -t:: show some brief memory usage statistics. --summary-first:: when
           multiple patchset diffs are being generated, put the patchset summary for all patchsets at the beginning of the
           output. --diffs-opts option string:: send a custom set of options to diff, for example to increase the number
           of context lines, or change the diff format. --debuglvl bitmask:: enable various debug output channels. -Z
           compression:: A value 1-9 which specifies amount of compression. A value of 0 disables compression. --root
           cvsroot:: Override the setting of CVSROOT (overrides working directory and environment). -i:: Incremental
           export. Each commit with no ancestor gets a from pointer name. When importing to an existing repository, this
           will attach each such commit as a child of the last commit on $BRANCH in the existing repository. -k:: Kill
           keywords: will extract files with -kk from the CVS archive to avoid noisy changesets. -T:: Force deterministic
           dates for regression testing. Each patchset will have a monotonic-increasing attributed date computed from its
           patchset ID. --fast-export:: Emit the report as a git import stream. --convert-ignores:: Convert ..cvsignore
           files to .gitignore files. --reposurgeon:: Emit for each commit a list of the CVS file:revision pairs composing
           it as a bzr-style commit property named "cvs-revisions". From version 2.12 onward, reposurgeon can interpret
           these and use them as hints for reference-lifting. -V:: Emit the program version and exit.  module-path::
           Operate on the specified module. If this option is not given, either the CVSROOT environment variable must be
           set to point directly at the module or cvsps must be run in a checkout directory or repository module
           subdirectory.

--
 

Re: State of CVS-to-git conversion tools (Was: Re: cvsps: bad usage: invalid argument --norc)

[Eric S. Raymond]

Ilya Basin <basinilya@gmail.com>:
> For new branches the 'from' command can refer the common ancestor in
> an existing branch. For example:
> 
>          /----------E thebranch
>         /
>     A---B---C---D master
> 
> Commit E is newer than D; we already imported D; thebranch is new.
> Instead of:
>     from refs/heads/thebranch^0
> refer the parent as:
>     from refs/heads/master^2

Understood.  Do you actually need this much generality in practice, 
or is it a theoretical case?

> OK, something's wrong with the man page: starting with '-A' the
> description is unstructured:

Interesting.  The aciidoc parser got a little confused, but inserting
some blank lines fixed it. 
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

Re[2]: State of CVS-to-git conversion tools (Was: Re: cvsps: bad usage: invalid argument --norc)

[Ilya Basin]

ESR> Ilya Basin <basinilya@gmail.com>:
>> For new branches the 'from' command can refer the common ancestor in
>> an existing branch. For example:
>> 
>>          /----------E thebranch
>>         /
>>     A---B---C---D master
>> 
>> Commit E is newer than D; we already imported D; thebranch is new.
>> Instead of:
>>     from refs/heads/thebranch^0
>> refer the parent as:
>>     from refs/heads/master^2

ESR> Understood.  Do you actually need this much generality in practice, 
ESR> or is it a theoretical case?

Such cases occur quite often.

--