[parisc-linux] CVS rumors

[parisc-linux] CVS rumors

[Paul Bame]

I've seen/heard some rumors which I'd like to clarify so we can make
good decisions rather than automatically assuming we need to drop to
assembly language because CVS is totally broken.

Rumors:
    1) date-based CVS checkout is hosed
    2) vendor branches are broken
    3) safe-cvsimport posts all changes to the trunk/head immediately

		date-based CVS checkout is hosed
		--------------------------------

Date-based CVS checkout *of the main branch* is indeed broken in one
(frequent) case and I filed it upstream and have heard nothing.  The test
case I filed has nothing to do with branches, vendor branches, safe-cvsimport,
client/server, nor even 'cvs import'.

This is a problem for us because we should be able to go back
to an older linux tree with this command:

	cvs up -D"some-date" -rHEAD

Currently this causes all the files which came from the initial linux
source import and which have never been changed.... to DISAPPEAR!  How
handy!  One work around is to add a gratuitous 1.2 revision to all those
files whose date is the same as the 1.1 revision -- most easily done by
hacking the RCS files.

BTW the following is ambiguous and probably shouldn't be allowed by CVS
if you have any branches:

	cvs up -D"some-date"

That grabs the files as of "some-date" from whatever arbitrary branch
they happen to be on -- thus you get some mixture of all the branches!

		    vendor branches are broken
		    --------------------------

Vendor branches are a special use of normal branches, and
*according to the CVS design* neither are "broken".  However the way
they are managed, mostly with 'cvs import', causes horrid problems
with certain use models.  The use model which works is that every time
you import from upstream, you immediately merge the upstream changes.
This is because the trunk/head is frequently BROKEN after 'cvs import'
so you *must* merge :-)  Here's why:

If you haven't changed a file on the trunk, and it changes upstream,
cvs import makes that change visible on the trunk immediately.  So for
example let's say you've modified a C file which implements an interface
defined in a header file.  You haven't modified the header file.  The
upstream folks modified the header file (and probably the C file too).
After cvs import your *trunk* has the new header file.  Your compile
will now probably fail.  Worse, this extends to files which were added
or removed upstream -- so the header could *disappear* after the import.
You can make up your own scenarios.

This problem is caused by 'cvs import' and has nothing to do with
CVS branches per se.

The way to cause REALLY BIG problems with vendor branches is to use them
for some upstream updates, but merge others by hand on to the trunk.  This
cuts the CVS design off at the knees!  We did this in our linux tree way
back and the number of problems is going down but isn't gone yet.

    safe-cvsimport posts all changes to the trunk/head immediately
    --------------------------------------------------------------

Re: [parisc-linux] CVS rumors

[Matthew Wilcox]

On Tue, Jul 10, 2001 at 09:08:26AM -0600, Paul Bame wrote:
> This is a problem for us because we should be able to go back
> to an older linux tree with this command:
> 
> 	cvs up -D"some-date" -rHEAD

The manpage doesn't indicate this is a valid use; indeed the wording
in some cases indicates that -D and -r are exclusive!

> 	cvs up -D"some-date"
> 
> That grabs the files as of "some-date" from whatever arbitrary branch
> they happen to be on -- thus you get some mixture of all the branches!

Yes, I noticed that.  But we have a real problem now, and I don't know how
to solve it.  Lamont wanted to check out version -pa30 -- how can he do that?

> Vendor branches are a special use of normal branches, and
> *according to the CVS design* neither are "broken".  However the way
> they are managed, mostly with 'cvs import', causes horrid problems
> with certain use models. 

Right, they're broken by design :-)

> This problem is caused by 'cvs import' and has nothing to do with
> CVS branches per se.
> 
> The way to cause REALLY BIG problems with vendor branches is to use them
> for some upstream updates, but merge others by hand on to the trunk.  This
> cuts the CVS design off at the knees!  We did this in our linux tree way
> back and the number of problems is going down but isn't gone yet.

I don't understand why we're using vendor branches at all though.  When
mang was doing CVS God duties, he imported Linus' stuff on an ordinary
branch.  What is the advantage of using vendor branches over ordinary ones?

> For now I'll be
> happy to consult and/or fix problems which result from safe-cvsimport.

No offence, but every time we've done a new import, we've discovered new
and excitingly different problems which have taken some time to be fixed.
I'm extremely nervous about using it while you're gone.

-- 
Revolutions do not require corporate support.

Re: [parisc-linux] CVS rumors

[Paul Bame]

= > 	cvs up -D"some-date" -rHEAD
= 
= The manpage doesn't indicate this is a valid use; indeed the wording
= in some cases indicates that -D and -r are exclusive!

yeah go figure :-(  semantically though, a date-based checkout must
add a branch for disambiguation, and except in this (frequent for us)
case it works.

= > 	cvs up -D"some-date"
= > 
= > That grabs the files as of "some-date" from whatever arbitrary branch
= > they happen to be on -- thus you get some mixture of all the branches!
= 
= Yes, I noticed that.  But we have a real problem now, and I don't know how
= to solve it.  Lamont wanted to check out version -pa30 -- how can he do that?

CM best practices usually involve explicitly tagging any release one
hopes to reproduce in the future and we could start doing that (I recommend
it).  For now either we apply the aforementioned workaround so we can
do untagged date-based checkouts, or I can come up with a procedure for
grabbing a suitable set of bits.

= > This problem is caused by 'cvs import' and has nothing to do with
= > CVS branches per se.
= > 
= > The way to cause REALLY BIG problems with vendor branches is to use them
= > for some upstream updates, but merge others by hand on to the trunk.  This
= > cuts the CVS design off at the knees!  We did this in our linux tree way
= > back and the number of problems is going down but isn't gone yet.
= 
= I don't understand why we're using vendor branches at all though.  When
= mang was doing CVS God duties, he imported Linus' stuff on an ordinary
= branch.  What is the advantage of using vendor branches over ordinary ones?

Vendor branches are both a concept and an implementation.  The
implementation sucks but the concept -- keeping clean upstream releases
on a separate branch -- to me is a good one and is one of the things
CVS does quite well.

safe-cvsimport essentially doesn't use the vendor branch *implementation*
(aka, how 'cvs import' would do it) but it does use the vendor branch
*concept*.  So safe-cvsimport is attempting to automate what mang did
by hand.

As it happens, there were corner cases which 'cvs import'
actually handled correctly which doing the same thing by hand on a branch
handled incorrectly, and that's the main difference between maintaining
an upstream vendor branch "by hand", and using safe-cvsimport.  I'm
considering rewriting safe-cvsimport to avoid 'cvs import' and live
with the unlikely corner case.

= > For now I'll be
= > happy to consult and/or fix problems which result from safe-cvsimport.
= 
= No offence, but every time we've done a new import, we've discovered new
= and excitingly different problems which have taken some time to be fixed.

So every time you've been aware of an import it has been problematic.  Maybe
you forgot a linux import *and merge* which I managed in a few hours?
I think perhaps there was a very quick glibc update too.  Maybe
you are unaware of the many times more imports I've done in other places
which are usually 100% successful?

When there are problems they require the same sorts of skills and repairs
as would "by hand" imports on the branch, though
usually I take a bit longer because I want to understand and fix the
problem rather than merely fixing the symptom.  I suspect the time spent
fixing safe-cvsimport and archive confusions is not "extra", but merely
seems/looks different partly because of the expectation that safe-cvsimport
*should* do all the right stuff automatically in a few minutes, in contrast
to the expectation that doing things by hand on a branch is a considerably
longer and more arduous task.

= I'm extremely nervous about using it while you're gone.

In the linux tree I echo your concern and there's really no way to be
rid of the vestiges of upstream imports which came into the trunk unless
we reinitialize our archive.  That plus the current brokenness of not
removing files which were removed upstream makes me think that
unless I do future imports, they might better be done "by
hand" on the "linus" branch, which I call a vendor branch though it
is not a cvs-style-broken-vendor-branch any longer.

	-P

Re: [parisc-linux] CVS rumors

[Michael Ang]

Paul Bame wrote:

>
> CM best practices usually involve explicitly tagging any release one
> hopes to reproduce in the future and we could start doing that (I recommend
> it).  For now either we apply the aforementioned workaround so we can
> do untagged date-based checkouts, or I can come up with a procedure for
> grabbing a suitable set of bits.


Tags are cheap.  Explicitly tagging at important moments is the way to 
go.  Relying on a date-based checkout is potentially less accurate, so 
IMO this shouldn't be the common practice.  There's no harm in adding a 
static tag, and you can always remove it or possibly fix it up if you 
get it wrong.


> = I don't understand why we're using vendor branches at all though.  When
> = mang was doing CVS God duties, he imported Linus' stuff on an ordinary
> = branch.  What is the advantage of using vendor branches over ordinary ones?
> 
> Vendor branches are both a concept and an implementation.  The
> implementation sucks but the concept -- keeping clean upstream releases
> on a separate branch -- to me is a good one and is one of the things
> CVS does quite well.
> 
> safe-cvsimport essentially doesn't use the vendor branch *implementation*
> (aka, how 'cvs import' would do it) but it does use the vendor branch
> *concept*.  So safe-cvsimport is attempting to automate what mang did
> by hand.


Writing an import script was one of those things I always meant to do in 
my copious spare time.  I'm glad to see that someone is actually doing 
the work :)  Where does the code for safe-cvsimport live?


> = > For now I'll be
> = > happy to consult and/or fix problems which result from safe-cvsimport.
> = 
> = No offence, but every time we've done a new import, we've discovered new
> = and excitingly different problems which have taken some time to be fixed.


All the more reason to roll the solutions found to these problems into a 
script.  Proper importing by hand is rather tedious and prone to 
operator error.  Also, any extra time spent fully understanding the 
problems and finding a robust solution will be regained over the 
following merges or if the cvs wrangler moves on to a different project 
-- hypothetically, of course ;)

> = I'm extremely nervous about using it while you're gone.
> 
> In the linux tree I echo your concern and there's really no way to be
> rid of the vestiges of upstream imports which came into the trunk unless
> we reinitialize our archive.  That plus the current brokenness of not
> removing files which were removed upstream makes me think that
> unless I do future imports, they might better be done "by
> hand" on the "linus" branch, which I call a vendor branch though it
> is not a cvs-style-broken-vendor-branch any longer.


I'd avoid using the term "vendor branch", since that already has a 
specific meaning to savvy CVS users (who will rightly criticize their 
use).  Calling it something like the "upstream branch" would avoid the 
overloaded meaning.  I haven't been following things enough to know what 
the issues related to "the vestiges of upstream imports" are.

	- Mike.

Re: [parisc-linux] CVS rumors

[Paul Bame]

= > CM best practices usually involve explicitly tagging any release one
= > hopes to reproduce in the future and we could start doing that (I recommend
= > it).  For now either we apply the aforementioned workaround so we can
= > do untagged date-based checkouts, or I can come up with a procedure for
= > grabbing a suitable set of bits.
= 
= Tags are cheap.  Explicitly tagging at important moments is the way to 
= go.  Relying on a date-based checkout is potentially less accurate, so 
= IMO this shouldn't be the common practice.  There's no harm in adding a 
= static tag, and you can always remove it or possibly fix it up if you 
= get it wrong.

Tagging a linux source tree over the network is slow however.

More information on date+branch CVS checkouts:  When there are multiple
branches, a date-based checkout must also supply a branch, implicitly
or explicitly, to disambiguate.  RCS has this feature
and it works fine (see the 'co' man page).  I can't force CVS to do it
though, on the trunk anyway, despite it's being built upon RCS :-(

So the workaround for date-based checkout of our trunk is.... use RCS
on a copy (can be 'cp -l') of our CVS repository :-( :-(

= Writing an import script was one of those things I always meant to do in 
= my copious spare time.  I'm glad to see that someone is actually doing 
= the work :)  Where does the code for safe-cvsimport live?

http://puffin.external.hp.com/cvs/build-tools/safe-cvsimport

Beware -- it uses 'cvs admin -b' plus at the moment seems not to remove
upstream-removed files correctly.  I'm thinking of re-doing it to
avoid using 'cvs import' altogether -- it is in need of a rewrite.

= I'd avoid using the term "vendor branch", since that already has a 
= specific meaning to savvy CVS users (who will rightly criticize their 
= use).  Calling it something like the "upstream branch" would avoid the 
= overloaded meaning.

Good idea.

= I haven't been following things enough to know what 
= the issues related to "the vestiges of upstream imports" are.

At least one upstream import+merge was done directly to the trunk.  So
changes which came with that import are unresolvable by
CVS during a merge and can require some sleuthing -- particularly
any files which were added/deleted at that time.

	-P

Re: [parisc-linux] CVS rumors

[Michael Ang]

Paul Bame wrote:


 > = Tags are cheap.  Explicitly tagging at important moments is the way to
 > = go.  Relying on a date-based checkout is potentially less accurate, so
 > = IMO this shouldn't be the common practice.  There's no harm in 
adding a
 > = static tag, and you can always remove it or possibly fix it up if you
 > = get it wrong.
 >
 > Tagging a linux source tree over the network is slow however.


True, but in the absence of a working date + branch checkout, it's a
reasonable interim solution (real solution is to fix cvs).


 > More information on date+branch CVS checkouts:  When there are multiple
 > branches, a date-based checkout must also supply a branch, implicitly
 > or explicitly, to disambiguate.  RCS has this feature
 > and it works fine (see the 'co' man page).  I can't force CVS to do it
 > though, on the trunk anyway, despite it's being built upon RCS :-(
 >
 > So the workaround for date-based checkout of our trunk is.... use RCS
 > on a copy (can be 'cp -l') of our CVS repository :-( :-(


Ugh.


 > = Where does the code for safe-cvsimport live?
 >
 > http://puffin.external.hp.com/cvs/build-tools/safe-cvsimport
 >
 > Beware -- it uses 'cvs admin -b' plus at the moment seems not to remove
 > upstream-removed files correctly.  I'm thinking of re-doing it to
 > avoid using 'cvs import' altogether -- it is in need of a rewrite.


I don't know of any way to get rid of the vendor branch taint other than
using 'cvs admin -b' or (less preferably) rcs directly.


 > = I haven't been following things enough to know what
 > = the issues related to "the vestiges of upstream imports" are.
 >
 > At least one upstream import+merge was done directly to the trunk.  So
 > changes which came with that import are unresolvable by
 > CVS during a merge and can require some sleuthing -- particularly
 > any files which were added/deleted at that time.


Ouch.  The import should have some distinct tag, and I guess using a
date tag is the best you'll get for trying to determine the state prior
to the merge (which, according to Murphy's Law, I predict overlapped
regular development).  My best advice is "don't do that then".

	- Mike.

Re: [parisc-linux] CVS rumors

[Paul Bame]

= True, but in the absence of a working date + branch checkout, it's a
= reasonable interim solution (real solution is to fix cvs).

There's another "solution".  date+branch checkout works on other branches
just not on the trunk :-)  One idea I propose for discussion is moving
development from the trunk to a branch so in the future we could do
date+branch checkouts.  We'll soon have 2 development branches
(2.4.x and 2.5.x) so we'll all need to learn to use a few more CVS
features anyway.

As for fixing CVS, there was no reply to my cvs bugs e-mail containing
the ultra-simple test case showing that HEAD+date checkouts don't work.

= I don't know of any way to get rid of the vendor branch taint other than
= using 'cvs admin -b' or (less preferably) rcs directly.

Unfortunately if you're actually using the CVS default branch feature,
safe-cvsimport's 'cvs admin -b' will make you unhappy :-)

The other option is to never use cvs import -- rather late for
us though there are some arguments for archiving the old repository
and reinitializing -- especially just prior to 2.5 kernels.

= Ouch.  The import should have some distinct tag, and I guess using a
= date tag is the best you'll get for trying to determine the state prior
= to the merge (which, according to Murphy's Law, I predict overlapped
= regular development).  My best advice is "don't do that then".

Fortunately by diffing against Linus' tree, willy tends to catch the
fallout from the mixed imports which we don't catch while manually
resolving merges.

	-P