import determinism

import determinism

[Enrico Weigelt]

Hi folks,


I'm curious on how deterministic the imports (git-cvsimport and
git-svn) are. Suppose I close the same cvs repo twice (assuming
no write access in between), are the resulting object SHA-1's
the same ?


cu
-- 
----------------------------------------------------------------------
 Enrico Weigelt, metux IT service -- http://www.metux.de/

 phone:  +49 36207 519931  email: weigelt@metux.de
 mobile: +49 151 27565287  icq:   210169427         skype: nekrad666
----------------------------------------------------------------------
 Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme
----------------------------------------------------------------------

Re: import determinism

[Ævar Arnfjörð Bjarmason]

On Sun, Nov 7, 2010 at 21:25, Enrico Weigelt <weigelt@metux.de> wrote:
> I'm curious on how deterministic the imports (git-cvsimport and
> git-svn) are. Suppose I close the same cvs repo twice (assuming
> no write access in between), are the resulting object SHA-1's
> the same ?

No, because the committer dates will be different.

Re: import determinism

[Andreas Schwab]

Ævar Arnfjörð Bjarmason <avarab@gmail.com> writes:

> On Sun, Nov 7, 2010 at 21:25, Enrico Weigelt <weigelt@metux.de> wrote:
>> I'm curious on how deterministic the imports (git-cvsimport and
>> git-svn) are. Suppose I close the same cvs repo twice (assuming
>> no write access in between), are the resulting object SHA-1's
>> the same ?
>
> No, because the committer dates will be different.

The committer info in every commit will be the same as the author info,
so the repository conversion is completely deterministic.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: import determinism

[Enrico Weigelt]

* Andreas Schwab <schwab@linux-m68k.org> wrote:

> The committer info in every commit will be the same as the author info,
> so the repository conversion is completely deterministic.

Thanks, that helps a lot.

My point is that I'm running a buch of automatic mirrors and like
to get them clustered. In theory, it should be enough - for cvs - 
running git-cvsimport regularily on all nodes, right ?


cu
-- 
----------------------------------------------------------------------
 Enrico Weigelt, metux IT service -- http://www.metux.de/

 phone:  +49 36207 519931  email: weigelt@metux.de
 mobile: +49 151 27565287  icq:   210169427         skype: nekrad666
----------------------------------------------------------------------
 Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme
----------------------------------------------------------------------

Re: import determinism

[Martin Langhoff]

On Sun, Nov 7, 2010 at 4:01 PM, Andreas Schwab <schwab@linux-m68k.org> wrote:
> The committer info in every commit will be the same as the author info,
> so the repository conversion is completely deterministic.

For cvsimport, it is not deterministic. Given cvs'
ambiguous/buggy/inconsistent internal semantics around some
operations, cvsps makes educated guesses about what happened.

Later commits can affect those educated guesses.

IIRC, this affects the exact tree at the birth of a branch, and files
added on a branch (instead of on HEAD).

If you use cvsimport in 'incremental' runs you won't have this
problem. Whatever it decided to do, sticks.

I haven't worked in detail with Keith Packard's parsecvs but I assume
it has similar issues.

hth,


m
-- 
 martin.langhoff@gmail.com
 martin@laptop.org -- School Server Architect
 - ask interesting questions
 - don't get distracted with shiny stuff  - working code first
 - http://wiki.laptop.org/go/User:Martinlanghoff

Re: import determinism

[Andreas Schwab]

Martin Langhoff <martin.langhoff@gmail.com> writes:

> On Sun, Nov 7, 2010 at 4:01 PM, Andreas Schwab <schwab@linux-m68k.org> wrote:
>> The committer info in every commit will be the same as the author info,
>> so the repository conversion is completely deterministic.
>
> For cvsimport, it is not deterministic. Given cvs'
> ambiguous/buggy/inconsistent internal semantics around some
> operations, cvsps makes educated guesses about what happened.
>
> Later commits can affect those educated guesses.

The OP was assuming an unchanging repository.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: import determinism

[Enrico Weigelt]

* Andreas Schwab <schwab@linux-m68k.org> wrote:

> > For cvsimport, it is not deterministic. Given cvs'
> > ambiguous/buggy/inconsistent internal semantics around some
> > operations, cvsps makes educated guesses about what happened.
> >
> > Later commits can affect those educated guesses.
> 
> The OP was assuming an unchanging repository.

My assumption is:

* the original cvs repo will have later additions
  (so I'm incrementally importing)
* no commints (besides cvsimport) in the git mirror, but others
  for off from there
* the mirror could get lost in an desaster (no separate backup)
  and should be recreated afresh in that case.


The point behind this is: I'm running a growing number of cvs2git
mirrors and dont want to do full backups of them.


cu
-- 
----------------------------------------------------------------------
 Enrico Weigelt, metux IT service -- http://www.metux.de/

 phone:  +49 36207 519931  email: weigelt@metux.de
 mobile: +49 151 27565287  icq:   210169427         skype: nekrad666
----------------------------------------------------------------------
 Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme
----------------------------------------------------------------------

Re: import determinism

[Martin Langhoff]

Every full clone is a good backup. If nobody is doing a full clone...
the do back them up. That's my recommendation.


M

On Tuesday, November 9, 2010, Enrico Weigelt <weigelt@metux.de> wrote:
> * Andreas Schwab <schwab@linux-m68k.org> wrote:
>
>> > For cvsimport, it is not deterministic. Given cvs'
>> > ambiguous/buggy/inconsistent internal semantics around some
>> > operations, cvsps makes educated guesses about what happened.
>> >
>> > Later commits can affect those educated guesses.
>>
>> The OP was assuming an unchanging repository.
>
> My assumption is:
>
> * the original cvs repo will have later additions
>   (so I'm incrementally importing)
> * no commints (besides cvsimport) in the git mirror, but others
>   for off from there
> * the mirror could get lost in an desaster (no separate backup)
>   and should be recreated afresh in that case.
>
>
> The point behind this is: I'm running a growing number of cvs2git
> mirrors and dont want to do full backups of them.
>
>
> cu
> --
> ----------------------------------------------------------------------
>  Enrico Weigelt, metux IT service -- http://www.metux.de/
>
>  phone:  +49 36207 519931  email: weigelt@metux.de
>  mobile: +49 151 27565287  icq:   210169427         skype: nekrad666
> ----------------------------------------------------------------------
>  Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme
> ----------------------------------------------------------------------
> --
> To unsubscribe from this list: send the line "unsubscribe git" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

-- 
 martin.langhoff@gmail.com
 martin@laptop.org -- School Server Architect
 - ask interesting questions
 - don't get distracted with shiny stuff  - working code first
 - http://wiki.laptop.org/go/User:Martinlanghoff

Re: import determinism

[Enrico Weigelt]

* Martin Langhoff <martin.langhoff@gmail.com> wrote:
> Every full clone is a good backup. If nobody is doing a full clone...
> the do back them up. That's my recommendation.

Does the clone have all information necessary for the incremental
cvsimport runs ?


cu
-- 
----------------------------------------------------------------------
 Enrico Weigelt, metux IT service -- http://www.metux.de/

 phone:  +49 36207 519931  email: weigelt@metux.de
 mobile: +49 151 27565287  icq:   210169427         skype: nekrad666
----------------------------------------------------------------------
 Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme
----------------------------------------------------------------------

Re: import determinism

[Martin Langhoff]

On Wed, Nov 10, 2010 at 11:18 AM, Enrico Weigelt <weigelt@metux.de> wrote:
> * Martin Langhoff <martin.langhoff@gmail.com> wrote:
>> Every full clone is a good backup. If nobody is doing a full clone...
>> the do back them up. That's my recommendation.
>
> Does the clone have all information necessary for the incremental
> cvsimport runs ?

Yes -- all cvsimport tracks is the branches. No extra import-tracking
data is maintained. (Other importers may or may not keep additional
data, YMMV, etc)


m
-- 
 martin.langhoff@gmail.com
 martin@laptop.org -- School Server Architect
 - ask interesting questions
 - don't get distracted with shiny stuff  - working code first
 - http://wiki.laptop.org/go/User:Martinlanghoff

Re: import determinism

[Enrico Weigelt]

* Martin Langhoff <martin.langhoff@gmail.com> wrote:

> > Does the clone have all information necessary for the incremental
> > cvsimport runs ?
> 
> Yes -- all cvsimport tracks is the branches. No extra import-tracking
> data is maintained. (Other importers may or may not keep additional
> data, YMMV, etc)

That's good news. Now I'll have to sit down and code some little
cluster management tool (mirrors are getting more and more), or
does anyone already did that ? ...


cu
-- 
----------------------------------------------------------------------
 Enrico Weigelt, metux IT service -- http://www.metux.de/

 phone:  +49 36207 519931  email: weigelt@metux.de
 mobile: +49 151 27565287  icq:   210169427         skype: nekrad666
----------------------------------------------------------------------
 Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme
----------------------------------------------------------------------

Re: import determinism

[Michael Haggerty]

On 11/07/2010 09:25 PM, Enrico Weigelt wrote:
> I'm curious on how deterministic the imports (git-cvsimport and
> git-svn) are. Suppose I close the same cvs repo twice (assuming
> no write access in between), are the resulting object SHA-1's
> the same ?

On 11/09/2010 02:43 PM, Enrico Weigelt wrote:
> The point behind this is: I'm running a growing number of cvs2git
> mirrors and dont want to do full backups of them.

If you are using cvs2git, why are you asking about git-cvsimport and
git-svn?

No tool that imports from CVS or Subversion can make a blanket guarantee
about consistency across conversions because both CVS and SVN allow
retroactive changes to the project history.  For example:

* Both CVS and SVN allow commit messages and other metadata of old
commits to be changed

* CVS allows files to be added retroactively to tags and branches with
no timestamp indicating that the file was not part of the original tag.

* CVS allows old revisions to be "obsoleted" (i.e., expunged from history).

* In CVS it is common practice for people to muck about directly in the
repository, for example renaming *,v files.

So (in the general case) there is no way to guarantee that two
independent conversions will have consistent results for the overlapping
parts of their history.  And even incremental conversions will
necessarily have to decide between converting the current state of the
repository accurately and converting in a way that is consistent with
earlier conversions.

In practice, especially if you are willing to constrain what the CVS
users are allowed to do, the overlapping parts of two conversions should
usually be identical or at least very similar (with older history more
likely to be identical).  Perhaps an rsync-style backup would be smart
enough to copy only the changed part of the history without excluding
the possibility that there are retroactive changes between subsequent
conversions.

If you run two cvs2git conversions on *exactly* the same CVS repository,
then the results *should* be identical.  I have tried always to process
data in a defined order rather than, say, in filesystem or
hashmap-determined order.  But AFAIK this property has not been tested
and could easily be buggy if I overlooked some source of indeterminism
somewhere in the cvs2git code.

Michael

-- 
Michael Haggerty
mhagger@alum.mit.edu
http://softwareswirl.blogspot.com/

Re: import determinism

[Enrico Weigelt]

* Michael Haggerty <mhagger@alum.mit.edu> wrote:
> On 11/07/2010 09:25 PM, Enrico Weigelt wrote:
> > I'm curious on how deterministic the imports (git-cvsimport and
> > git-svn) are. Suppose I close the same cvs repo twice (assuming
> > no write access in between), are the resulting object SHA-1's
> > the same ?
> 
> On 11/09/2010 02:43 PM, Enrico Weigelt wrote:
> > The point behind this is: I'm running a growing number of cvs2git
> > mirrors and dont want to do full backups of them.
> 
> If you are using cvs2git, why are you asking about git-cvsimport and
> git-svn?

No, I'm not using the "cvs2git" tool (wasn't aware of that at all) ;-)
I'm using git-cvsimport.


cu
-- 
----------------------------------------------------------------------
 Enrico Weigelt, metux IT service -- http://www.metux.de/

 phone:  +49 36207 519931  email: weigelt@metux.de
 mobile: +49 151 27565287  icq:   210169427         skype: nekrad666
----------------------------------------------------------------------
 Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme
----------------------------------------------------------------------