[ANNOUNCE] BK->CVS (real time mirror)

[ANNOUNCE] BK->CVS (real time mirror)

[Larry McVoy]

We've been working on a gateway between BitKeeper and CVS to provide
the revision history in a form which makes the !BK people happy (or
happier).

We have the first pass of this completed and have a linux 2.5 tree on
kernel.bkbits.net and you can check out the tree as follows (please don't
do this unless you are a programmer and will be using this.  Penguin
Computing provided the hardware and the bandwidth for that machine and
if you all melt down the network they could get annoyed.  By all means
go for it if you actually write code, though, that's why it is there.)

    mkdir ws
    cd ws
    cvs -d:pserver:anonymous@kernel.bkbits.net:/home/cvs co linux-2.5

Each of the releases are tagged, they are of the form v2_5_64 etc.

Linus had said in the past that someone other than us should do this but
as it turns out, to do a reasonable job you need BK source.  So we did it.
What do we mean by a reasonable job?  BitKeeper has an automatic branch
feature which captures all parallel development.  It's cool but a bit
pedantic and it makes exporting to a different system almost impossible
if you try and match what BK does exactly.  So we didn't.  What we
(actually Wayne Scott) did was to write a graph traversal alg which
finds the longest path through the revision history which includes
all tags.  For the 2.5 tree, that is currently 8298 distinct points.
Each of those points has been captured in CVS as a commit.  If we did
our job correctly, each of these commits has the same timestamp across
all files.  So you should be able to get any changeset out of the CVS
tree with the appropriate CVS command based on dates.

We also created a ChangeSet file in the CVS tree.  It has no contents, it
serves as a place to capture the BK changeset comments.  Each file which
is part of a changeset has an extra comment which is of the form

	(Logical change 1.%d)

where the "1.%d" matches the changeset rev.  So you can look for all files
that have (Logical change 1.300) in their comments to reconstruct the 
changeset.  NOTE!  That information is actually redundant, the timestamps
are supposed to do the same thing, let us know if that is not working, we'll
redo it.  I expect we'll find bugs, please be patient, it takes 4 hours of
CPU time on a 2.1Ghz Athlon to do the conversion, that's a big part of 
why this has taken so long.  That's after a week's worth of optimizations.

Each ChangeSet delta has a BK rev associated with it in the comments.
We'll be giving you a small shell script which you can use to send Linus
patches that include the rev and we'll modify BK so that it can take
those patches with no patch rejects if you used that script.

We have a first pass of a real time gateway between BK and this CVS tree 
done.  Right now it is done by hand (by me) but as soon as it is debugged
you will see this tree being updated about 1-3 minutes after Linus pushes
to bkbits.  

Once you guys look this over and decide you like it, we'll do the same
thing for the 2.4 tree.

We're also talking to an unnamed (in case it doesn't work out) Linux
company who may host bkbits.net for us.  If they do that, we'll turn
the GNU patch exporter feature in BKD.  That means that you'll be able
to wget any changeset as a GNU patch, complete with checkin comments.
I'm working with Alan on the format, I think we're close though I have
to run the latest version past him.

If all of this sounds nice, it is.  It was a lot of work for us to do
this and you might be wondering why we bothered.  Well, for a couple of
reasons.  First of all, it was only recently that I realized that because
BK is not free software some people won't run BK to get data out of BK.
It may be dense on my part, but I simply did not anticipate that people
would be that extreme, it never occurred to me.  We did a ton of work to
make sure anyone could get their data out of BK but you do have to run
BK to get the data.  I never thought of people not being willing to run
BK to get at the data.  Second, we have maintained SCCS compatible file
formats so that there would be another way to get the data out of BK.
This has held us back in terms of functionality and performance.  I had
thought there was some value in the SCCS format but recent discussions
on this list have convinced me that without the changeset information
the file format doesn't have much value.

Our goal is to provide the data in a way that you can get at it without
being dependent on us or BK in any way.  As soon as we have this
debugged, I'd like to move the CVS repositories to kernel.org (if I can
get HPA to agree) and then you'll have the revision history and can live
without the fear of the "don't piss Larry off license".  Quite frankly,
we don't like the current situation any better than many of you, so if
this addresses your concerns that will take some pressure off of us.

Another goal is to have the freedom to evolve our file formats to be
better, better performance and more features.  SCCS is holding us back.
So you should look hard at what we are providing and figure out if it
is enough.  If you come back with "well, it's not BitKeeper so it's
not enough" we'll just ignore that.  CVS isn't BitKeeper.  On the
other hand, we believe we have gone as far as is possible to provide
all of the information, checkin comments, data, timestamps, user names,
everything.  The graph traversal alg captures information at an extremely
fine granularity, absolutely as fine is possible.  We have 8298 distinct
points over the 2.5.0 .. 2.5.64 set of changes, so it is 130 times finer
than the official releases.  If you think something is missing, tell us,
we'll try and fix it.

The payoff for you is that you have the data in a format that is not
locked into some tool which could be taken away.  The payoff for us is
that we can evolve our tool as we see fit.  We have that right today,
we can do whatever we want, but it would be anywhere from annoying
to unethical to do so if that meant that you couldn't get at the data
except through BitKeeper.  So the "deal" here is that you get the data
in CVS (and/or patches + comments) and we get to hack the heck out of
the file format.  Our changes are going to move far faster than CSSC or
anyone else could keep up without a lot of effort.  On the other hand,
our changes are going to make cold cache performance be much closer to
hot cache performance, use a lot less disk space, a lot less memory,
and a lot less CPU.

So take a look and tell me what you think.
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: [ANNOUNCE] BK->CVS (real time mirror)

[Ben Collins]

> The payoff for you is that you have the data in a format that is not
> locked into some tool which could be taken away.  The payoff for us is
> that we can evolve our tool as we see fit.  We have that right today,
> we can do whatever we want, but it would be anywhere from annoying
> to unethical to do so if that meant that you couldn't get at the data
> except through BitKeeper.  So the "deal" here is that you get the data
> in CVS (and/or patches + comments) and we get to hack the heck out of
> the file format.  Our changes are going to move far faster than CSSC or
> anyone else could keep up without a lot of effort.  On the other hand,
> our changes are going to make cold cache performance be much closer to
> hot cache performance, use a lot less disk space, a lot less memory,
> and a lot less CPU.

Larry, I don't mean to start yet another anti-bitmover, anti-bitkeeper or
anti-larry flame-fest, but I have to be honest that I am a little bit
worried.

You are giving us approximately 90% of our data in exchange for the one
thing that made using bitkeeper not a total sellout; the fact that the
revision history of the repo was still accessible without proprietary
software.

I honestly appreciate the work that you and BitMover do for the kernel,
but not giving us access to 100% of _our_ data is unacceptable to me.
Quite honestly, I think your move is to restrict the possible
alternatives to the BK client (the CSSC based ones like I and others had
done), which were able to extract 100% of the data, even if they
couldn't make use of it in the same way as bitkeeper. Atleast it was
there.

You've made quite a marketing move. It's obvious to me, maybe not to
others. By providing this CVS gateway, you make it almost pointless to
work on an alternative client. Also by providing it, you make it easier
to get away with locking the revision history into a proprietary format.



-- 
Debian     - http://www.debian.org/
Linux 1394 - http://www.linux1394.org/
Subversion - http://subversion.tigris.org/
Deqo       - http://www.deqo.com/

Re: [ANNOUNCE] BK->CVS (real time mirror)

[H. Peter Anvin]

Followup to:  <20030312034330.GA9324@work.bitmover.com>
By author:    Larry McVoy <lm@bitmover.com>
In newsgroup: linux.dev.kernel
> 
> If all of this sounds nice, it is.  It was a lot of work for us to do
> this and you might be wondering why we bothered.  Well, for a couple of
> reasons.  First of all, it was only recently that I realized that because
> BK is not free software some people won't run BK to get data out of BK.
> It may be dense on my part, but I simply did not anticipate that people
> would be that extreme, it never occurred to me.  We did a ton of work to
> make sure anyone could get their data out of BK but you do have to run
> BK to get the data.  I never thought of people not being willing to run
> BK to get at the data.  Second, we have maintained SCCS compatible file
> formats so that there would be another way to get the data out of BK.
> This has held us back in terms of functionality and performance.  I had
> thought there was some value in the SCCS format but recent discussions
> on this list have convinced me that without the changeset information
> the file format doesn't have much value.
> 

I can only speak for myself, but I didn't mind until the license ended
up having the "unless you hack on other tools" exception in it.
Personally, I value my freedom to hack on whatever I want a lot more
than the convenience of BK.  This is a personal choice on my part and
may sound "extreme" to you, and other people have made other
tradeoffs, but for me freedom was the reason I started hacking Linux
instead of becoming a Win32 geek.

Having this capability available will certainly make life better for
everyone involved.  Besides, "we won't hold your data hostage" is
actually a pretty nice selling argument.

> 
> Our goal is to provide the data in a way that you can get at it without
> being dependent on us or BK in any way.  As soon as we have this
> debugged, I'd like to move the CVS repositories to kernel.org (if I can
> get HPA to agree) and then you'll have the revision history and can live
> without the fear of the "don't piss Larry off license".  Quite frankly,
> we don't like the current situation any better than many of you, so if
> this addresses your concerns that will take some pressure off of us.
> 

I'm sure we can work something out.  However, at the moment
zeus.kernel.org, our main server with lots and lots of bandwidth, is
starting to run into its limits, so I can't promise *when* that will
happen.  Just putting in another server 

	-hpa
-- 
<hpa@transmeta.com> at work, <hpa@zytor.com> in private!
"Unix gives you enough rope to shoot yourself in the foot."
Architectures needed: ia64 m68k mips64 ppc ppc64 s390 s390x sh v850 x86-64

Re: [ANNOUNCE] BK->CVS (real time mirror)

[Larry McVoy]

On Tue, Mar 11, 2003 at 08:39:19PM -0800, H. Peter Anvin wrote:
> I can only speak for myself, but I didn't mind until the license ended
> up having the "unless you hack on other tools" exception in it.
> Personally, I value my freedom to hack on whatever I want a lot more
> than the convenience of BK.  

Yeah, that's cool, I don't blame you, it's a pretty extreme clause.
We may well drop it in the future if we feel we have pulled far enough
ahead that everyone else is just playing catchup.  I do apologize for
that clause, I know it caused a lot of concern, but try and remember that
I'm "you" in that I'm just an engineer figuring this stuff out as I go.
We try and fix it as we go as well so there is hope.

> Having this capability available will certainly make life better for
> everyone involved.  Besides, "we won't hold your data hostage" is
> actually a pretty nice selling argument.

Yup.  I really thought that all the export stuff was !hostage but I
didn't factor in the license issues.

> > As soon as we have this
> > debugged, I'd like to move the CVS repositories to kernel.org (if I can
> > get HPA to agree) 
> 
> I'm sure we can work something out.  However, at the moment
> zeus.kernel.org, our main server with lots and lots of bandwidth, is
> starting to run into its limits, so I can't promise *when* that will
> happen.  Just putting in another server 

We can certainly pay for a server, a server is not that much money and
is not an ongoing cost.  So if that's the problem, I can get you a server
in less than a week, ping me off line and we'll work it out.

The main thing is that the CVS server and the tarball of the CVS repository
are *not* under our control.  That's the only way some people are going to
believe that we're not out to screw them and it would oh-so-nice to have
people think that, it really would.
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: [ANNOUNCE] BK->CVS (real time mirror)

[Jens Axboe]

On Tue, Mar 11 2003, Ben Collins wrote:
> You've made quite a marketing move. It's obvious to me, maybe not to
> others. By providing this CVS gateway, you make it almost pointless to
> work on an alternative client. Also by providing it, you make it easier
> to get away with locking the revision history into a proprietary format.

This is a really good point, deserves high lighting imho...

The BK candy is getting increasingly bitter to swallow here, I may just
have to drop it soon. A shame.

-- 
Jens Axboe

Re: [ANNOUNCE] BK->CVS (real time mirror)

[Andreas Dilger]

On Mar 12, 2003  09:55 +0100, Jens Axboe wrote:
> On Tue, Mar 11 2003, Ben Collins wrote:
> > You've made quite a marketing move. It's obvious to me, maybe not to
> > others. By providing this CVS gateway, you make it almost pointless to
> > work on an alternative client. Also by providing it, you make it easier
> > to get away with locking the revision history into a proprietary format.
> 
> This is a really good point, deserves high lighting imho...
> 
> The BK candy is getting increasingly bitter to swallow here, I may just
> have to drop it soon. A shame.

Sadly, some people see the dark side of everything.  I don't see how making
a CVS repository available with comments and an as-good-as-you-can-do-with-CVS
equivalent of a BK changeset equals "locking the revision history into a
proprietary format".  Yes, Larry said that this would allow him to change the
BK file format to break compatibility with CSSC, but it is no more "locked
away" now than before for those people who refuse to use BK.

Ironically, SCCS was a former "evil proprietary format" that was reverse
engineered to get CSSC, AFAIK.  People are still free to update CSSC to
track BK if they so choose.

Some people will just never be happy no matter what you give them.

Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://www-mddsp.enel.ucalgary.ca/People/adilger/

Re: [ANNOUNCE] BK->CVS (real time mirror)

[Jens Axboe]

On Wed, Mar 12 2003, Andreas Dilger wrote:
> Some people will just never be happy no matter what you give them.

I've been very happy with BK, been using it shortly after Linus started
doing so. Mostly out of curiosity at first, later because it was
actually quite useful. I even see myself as a fairly pragmatic
individual, but even so I do find it increasingly difficult to defend my
BK usage.

So please stop thinking you can judge that easily by pushing me into
your nice little 'some people will never be happy bla bla' category.

-- 
Jens Axboe

Re: [ANNOUNCE] BK->CVS (real time mirror)

[Andreas Dilger]

On Mar 12, 2003  11:31 +0100, Jens Axboe wrote:
> I've been very happy with BK, been using it shortly after Linus started
> doing so. Mostly out of curiosity at first, later because it was
> actually quite useful. I even see myself as a fairly pragmatic
> individual, but even so I do find it increasingly difficult to defend my
> BK usage.

Interesting.  I _had_ lumped you into the "unhappy with BK" camp that has
become so vocal on l-k these days.  My apologies.  I do find it sort of sad
that you (or anyone) actually have to defend your BK usage to others.

I'm personally a "do what you want and let others do what they want as
long as it doesn't interfere with me" kind of person, but it seems that
lots of people here have the opinion that they know what is better for
everyone else, and have no problem telling the list over an over about it.
Probably time to fork a linux-code-repository mailing list and have everyone
spend their time over there instead of rehashing BK flamewars and/or BK
replacement here every week.

Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://www-mddsp.enel.ucalgary.ca/People/adilger/

Re: [ANNOUNCE] BK->CVS (real time mirror)

[Jens Axboe]

On Wed, Mar 12 2003, Andreas Dilger wrote:
> On Mar 12, 2003  11:31 +0100, Jens Axboe wrote:
> > I've been very happy with BK, been using it shortly after Linus started
> > doing so. Mostly out of curiosity at first, later because it was
> > actually quite useful. I even see myself as a fairly pragmatic
> > individual, but even so I do find it increasingly difficult to defend my
> > BK usage.
> 
> Interesting.  I _had_ lumped you into the "unhappy with BK" camp that has
> become so vocal on l-k these days.  My apologies.  I do find it sort of sad
> that you (or anyone) actually have to defend your BK usage to others.

No offense taken, and I personally don't have any sort of political
agenda that I care to voice on lkml :). That's part of where the
pragmatism comes in, I just don't care enough.

About every patch I sent here on lkml has been generated by bk for the
past year. I typically don't do commits, just have trees with pending
deltas and bk -r diffs -u does the job for me.

> I'm personally a "do what you want and let others do what they want as
> long as it doesn't interfere with me" kind of person, but it seems that
> lots of people here have the opinion that they know what is better for
> everyone else, and have no problem telling the list over an over about it.
> Probably time to fork a linux-code-repository mailing list and have everyone
> spend their time over there instead of rehashing BK flamewars and/or BK
> replacement here every week.

For me, I think Andrew's patch management scripts will do the job. And
yeah, the non-stop bk threads make me sick as well and are rarely read
here. So I better make this my last mail on the subject...

-- 
Jens Axboe

Re: [ANNOUNCE] BK->CVS (real time mirror)

[Jamie Lokier]

Andreas Dilger wrote:
> Ironically, SCCS was a former "evil proprietary format" that was reverse
> engineered to get CSSC, AFAIK.  People are still free to update CSSC to
> track BK if they so choose.

Actually the SCCS format is documented in a manual page (although
there are a few annoying ambiguities in it).

-- Jamie

Re: [ANNOUNCE] BK->CVS (real time mirror)

[H. Peter Anvin]

Followup to:  <20030312032614.G12806@schatzie.adilger.int>
By author:    Andreas Dilger <adilger@clusterfs.com>
In newsgroup: linux.dev.kernel
> 
> Sadly, some people see the dark side of everything.  I don't see how making
> a CVS repository available with comments and an as-good-as-you-can-do-with-CVS
> equivalent of a BK changeset equals "locking the revision history into a
> proprietary format".  Yes, Larry said that this would allow him to change the
> BK file format to break compatibility with CSSC, but it is no more "locked
> away" now than before for those people who refuse to use BK.
> 
> Ironically, SCCS was a former "evil proprietary format" that was reverse
> engineered to get CSSC, AFAIK.  People are still free to update CSSC to
> track BK if they so choose.
> 
> Some people will just never be happy no matter what you give them.
> 

>From what I can gather, the question is very simple:

"Can we get our data out of BK into some kind of open format?"

It's an important question.  If the answer is "yes, but only the stuff
that can be mapped onto CVS" then that's a significant data loss, and
if BitMover changes the data format without documentation, then there
is no longer a way to get all the data out.

Presumably the CVS exporter could get augmented with some kind of
metadata export... perhaps an XML schema that describes how the
various points are to be linked or whatnot... it won't turn CVS into
BK overnight (so Larry can still sleep at night), but it would give
BitMover the freedom to change their data format.

	-hpa
-- 
<hpa@transmeta.com> at work, <hpa@zytor.com> in private!
"Unix gives you enough rope to shoot yourself in the foot."
Architectures needed: ia64 m68k mips64 ppc ppc64 s390 s390x sh v850 x86-64

Re: [ANNOUNCE] BK->CVS (real time mirror)

[Ben Collins]

On Wed, Mar 12, 2003 at 03:26:14AM -0700, Andreas Dilger wrote:
> On Mar 12, 2003  09:55 +0100, Jens Axboe wrote:
> > On Tue, Mar 11 2003, Ben Collins wrote:
> > > You've made quite a marketing move. It's obvious to me, maybe not to
> > > others. By providing this CVS gateway, you make it almost pointless to
> > > work on an alternative client. Also by providing it, you make it easier
> > > to get away with locking the revision history into a proprietary format.
> > 
> > This is a really good point, deserves high lighting imho...
> > 
> > The BK candy is getting increasingly bitter to swallow here, I may just
> > have to drop it soon. A shame.
> 
> Sadly, some people see the dark side of everything.  I don't see how making
> a CVS repository available with comments and an as-good-as-you-can-do-with-CVS
> equivalent of a BK changeset equals "locking the revision history into a
> proprietary format".  Yes, Larry said that this would allow him to change the
> BK file format to break compatibility with CSSC, but it is no more "locked
> away" now than before for those people who refuse to use BK.
> 
> Ironically, SCCS was a former "evil proprietary format" that was reverse
> engineered to get CSSC, AFAIK.  People are still free to update CSSC to
> track BK if they so choose.

Atleast SCCS is mostly ascii. Larry is talking about binary. Who knows,
maybe even encrypted and using some unknown compression method (I'm sure
if it's encrypted, it will be called "compression").


-- 
Debian     - http://www.debian.org/
Linux 1394 - http://www.linux1394.org/
Subversion - http://subversion.tigris.org/
Deqo       - http://www.deqo.com/

Re: [ANNOUNCE] BK->CVS (real time mirror)

[Dana Lacoste]

On Wed, 2003-03-12 at 11:13, H. Peter Anvin wrote:
> "Can we get our data out of BK into some kind of open format?"

> It's an important question.  If the answer is "yes, but only the stuff
> that can be mapped onto CVS" then that's a significant data loss, and
> if BitMover changes the data format without documentation, then there
> is no longer a way to get all the data out.

This sounds like the old GPL argument.

The GPL'd redistributor has to supply the source, they don't have to
supply it in the format that's best for you, being an 80mm tape drive
cuz you're stuck in the punch card age.

Seriously, if CVS loses all that data, is that BK's fault?  BK's so
powerful because it has more information than anyone else, but it's
not their fault (and it's not proprietary data) that no-one else can
deal with the data when it's exported, now is it????

It's not a significant data loss when you try to view a 24bpp image
on an 8bpp display, so it's not a significant data loss that CVS can't
handle the BK.  If it could, Linus would've switched to CVS instead....

I'm not saying Larry's a God or anything, I'm just hoping you guys can
give it up already.  Linus uses BK, nobody else needs to, so move on!

Dana Lacoste
Ottawa, Canada

Re: [ANNOUNCE] BK->CVS (real time mirror)

[John Bradford]

> Linus uses BK, nobody else needs to, so move on!

Seconded.

John.

Re: [ANNOUNCE] BK->CVS (real time mirror)

[Lars Marowsky-Bree]

On 2003-03-12T11:18:38,
   Ben Collins <bcollins@debian.org> said:

> Atleast SCCS is mostly ascii. Larry is talking about binary. Who knows,
> maybe even encrypted and using some unknown compression method (I'm sure
> if it's encrypted, it will be called "compression").

*sigh* However, all Larry _could_ be talking about is that he wants to replace
the SCCS format by something more powerful. Nowhere did he say that the format
would not be documented.

Granted, he also did not say that it _would_ be, but you all are jumping so
hard on him based on the assumption that it would not be without knowing that
either, so maybe you could have just written to Larry and asked?

I'm rather agnostic to the BK debate: I think it is an awesome tool, and if it
gets the job done, I am all for it. If you don't want to use it for whatever
reason, that's fine too. And asking for the Linux Kernel data to be fully
available without using a proprietary tool also makes lots of sense. I also
agree with Larry that duplicating the work done in BK in an Open Source tool
is going to take quite a while and effort: I'm not going as far as saying that
it cannot be done, because Linux itself is the best example that it _can_ be
done if people really want too. But if you want, more power to you, too.

But the 'BK and Larry are evil to the bone because it is not GPL!' crap is
annoying the hell out of me. Shut up, will you all, pretty please? And could
you please first ask/clarify, then flame?


Sincerely,
    Lars Marowsky-Brée <lmb@suse.de>

-- 
SuSE Labs - Research & Development, SuSE Linux AG
  
"If anything can go wrong, it will." "Chance favors the prepared (mind)."
  -- Capt. Edward A. Murphy            -- Louis Pasteur

Re: [ANNOUNCE] BK->CVS (real time mirror)

[Roman Zippel]

Hi,

On 12 Mar 2003, Dana Lacoste wrote:

> Seriously, if CVS loses all that data, is that BK's fault?  BK's so
> powerful because it has more information than anyone else, but it's
> not their fault (and it's not proprietary data) that no-one else can
> deal with the data when it's exported, now is it????

That's not the point. Larry does not own the data in the Linux repository, 
this was one of the conditions for the bk usage, so Larry cannot say, that 
you only get all the data if you use bk. If cvs can't represent all the 
information, we have to find another solution.

bye, Roman

Re: [ANNOUNCE] BK->CVS (real time mirror)

[H. Peter Anvin]

Dana Lacoste wrote:
> On Wed, 2003-03-12 at 11:13, H. Peter Anvin wrote:
> 
>>"Can we get our data out of BK into some kind of open format?"
> 
> 
>>It's an important question.  If the answer is "yes, but only the stuff
>>that can be mapped onto CVS" then that's a significant data loss, and
>>if BitMover changes the data format without documentation, then there
>>is no longer a way to get all the data out.
> 
> 
> This sounds like the old GPL argument.
> 
> The GPL'd redistributor has to supply the source, they don't have to
> supply it in the format that's best for you, being an 80mm tape drive
> cuz you're stuck in the punch card age.
> 
> Seriously, if CVS loses all that data, is that BK's fault?  BK's so
> powerful because it has more information than anyone else, but it's
> not their fault (and it's not proprietary data) that no-one else can
> deal with the data when it's exported, now is it????
> 
> It's not a significant data loss when you try to view a 24bpp image
> on an 8bpp display, so it's not a significant data loss that CVS can't
> handle the BK.  If it could, Linus would've switched to CVS instead....
> 

You're missing the point completely.

Of course it's not BK's fault that CVS can't represent the data. 
However, one of the (valid!) selling points of BK was "we won't hold 
your data hostage."  That requires that you can export both the data and 
the metadata into some kind of open format.  Since CVS clearly can't be 
that open format (CVS being insufficiently powerful), the additional 
metadata needs to be available in some kind of auxilliary form.  It's 
then, of course, not BK's fault that CVS can't possibly make use of that 
auxilliary metadata.

	-hpa

Re: [ANNOUNCE] BK->CVS (real time mirror)

[Ryan Anderson]

On Wed, Mar 12, 2003 at 05:47:41PM +0100, Lars Marowsky-Bree wrote:
> On 2003-03-12T11:18:38,
>    Ben Collins <bcollins@debian.org> said:
> 
> > Atleast SCCS is mostly ascii. Larry is talking about binary. Who knows,
> > maybe even encrypted and using some unknown compression method (I'm sure
> > if it's encrypted, it will be called "compression").
> 
> *sigh* However, all Larry _could_ be talking about is that he wants to replace
> the SCCS format by something more powerful. Nowhere did he say that the format
> would not be documented.
> 
> Granted, he also did not say that it _would_ be, but you all are jumping so
> hard on him based on the assumption that it would not be without knowing that
> either, so maybe you could have just written to Larry and asked?

This is, I think, an underlying assumption that people are making.

Larry, given that I *think* you've said that the algorithms are the
important parts, not the file format (at least in the past), would you
consider publicly documenting the file format?

Thanks for the CVS gateway, anyway, though.

-- 

Ryan Anderson
  sometimes Pug Majere

Re: [ANNOUNCE] BK->CVS (real time mirror)

[John Bradford]

> > This sounds like the old GPL argument.
> > 
> > The GPL'd redistributor has to supply the source, they don't have to
> > supply it in the format that's best for you, being an 80mm tape drive
> > cuz you're stuck in the punch card age.
> > 
> > Seriously, if CVS loses all that data, is that BK's fault?  BK's so
> > powerful because it has more information than anyone else, but it's
> > not their fault (and it's not proprietary data) that no-one else can
> > deal with the data when it's exported, now is it????
> > 
> > It's not a significant data loss when you try to view a 24bpp image
> > on an 8bpp display, so it's not a significant data loss that CVS can't
> > handle the BK.  If it could, Linus would've switched to CVS instead....
> > 
> 
> You're missing the point completely.
> 
> Of course it's not BK's fault that CVS can't represent the data. 
> However, one of the (valid!) selling points of BK was "we won't hold 
> your data hostage."  That requires that you can export both the data and 
> the metadata into some kind of open format.  Since CVS clearly can't be 
> that open format (CVS being insufficiently powerful), the additional 
> metadata needs to be available in some kind of auxilliary form.  It's 
> then, of course, not BK's fault that CVS can't possibly make use of that 
> auxilliary metadata.

I thought that BK has been able to export everything to a text file
since the first version.

(Ah, but of course, unless that text file is available in EBCDIC, we
still have a problem...)

John.

Re: [ANNOUNCE] BK->CVS (real time mirror)

[Larry McVoy]

> I thought that BK has been able to export everything to a text file
> since the first version.

bk export -tpatch -r1.900 > patch.1.900
bk changes -v -r1.900 > comments.1.900

Been there forever.  So has ways to get all the metadata from the command
line without having to reverse engineer the file format.  See

    http://www.bitkeeper.com/manpages/bk-prs-1.html

it's all there.  Always has been.

Wayne wanted me to point that it is easy to write the BK to CVS exporter
completely from the command line, we prototyped it that way, the only
reason we rewrote part of it in C was for performance.  The point being
that you guys could have done this yourself without help from us because
all the metadata is right there.  Ditto for anyone else worried about 
getting their data out of BK now or in the future.  The whole point of
prs is to be able to have a will-always-work way to get at the data or
the metadata, it makes the file format a non-issue.  
-- 
---
Larry McVoy              lm at bitmover.com          http://www.bitmover.com/lm

Re: [ANNOUNCE] BK->CVS (real time mirror)

[Arador]

On Tue, 11 Mar 2003 23:16:21 -0500
Ben Collins <bcollins@debian.org> wrote:

> You've made quite a marketing move. It's obvious to me, maybe not to
> others. By providing this CVS gateway, you make it almost pointless to
> work on an alternative client. Also by providing it, you make it easier

I don't think so. This also bits Larry. If he does well enought, there'll be
some people here that won't use bitkeeper just because they can use the cvs
gateway and they don't need/miss the features they could get with bk.

And i don't think it avoids creating a free bk clone. I guess that there's
a lot of people out there interested in such tool, and not only for kernel
development; this won't stop them.

As far as i can see; Larry is just wasting time (money) to help the kernel
development and people who doesn't use BK just because it isn't free. And
he's not charging me, so i find this a good movement for everybody. I only
can say thanks.


Diego Calleja

Re: [ANNOUNCE] BK->CVS (real time mirror)

[Ben Collins]

On Wed, Mar 12, 2003 at 07:38:06PM +0100, Arador wrote:
> On Tue, 11 Mar 2003 23:16:21 -0500
> Ben Collins <bcollins@debian.org> wrote:
> 
> > You've made quite a marketing move. It's obvious to me, maybe not to
> > others. By providing this CVS gateway, you make it almost pointless to
> > work on an alternative client. Also by providing it, you make it easier
> 
> I don't think so. This also bits Larry. If he does well enought, there'll be
> some people here that won't use bitkeeper just because they can use the cvs
> gateway and they don't need/miss the features they could get with bk.
> 
> And i don't think it avoids creating a free bk clone. I guess that there's
> a lot of people out there interested in such tool, and not only for kernel
> development; this won't stop them.
> 
> As far as i can see; Larry is just wasting time (money) to help the kernel
> development and people who doesn't use BK just because it isn't free. And
> he's not charging me, so i find this a good movement for everybody. I only
> can say thanks.

You're missing the point. I am not against the CVS->BK gateway. I'm all
for it. But it's kind of sour given that he now wants to change the disk
format of the repo to make it harder to get the data from it.

If all he announced was "you now have a CVS->BK repo", I wouldn't be
complaining, I'd be patting him on the back.



-- 
Debian     - http://www.debian.org/
Linux 1394 - http://www.linux1394.org/
Subversion - http://subversion.tigris.org/
Deqo       - http://www.deqo.com/

Re: [ANNOUNCE] BK->CVS (real time mirror)

[Andreas Dilger]

On Mar 12, 2003  13:47 -0500, Ben Collins wrote:
> You're missing the point. I am not against the CVS->BK gateway. I'm all
> for it. But it's kind of sour given that he now wants to change the disk
> format of the repo to make it harder to get the data from it.

Actually, that is purely YOU reading something into what he wrote.  He
didn't say "now I'm going to make the repo harder to get data from it".
What he said was "now I'm free to change the format from SCCS to something
that is more efficient for BK to use".  Who knows, maybe the new format
will be _easier_ to reverse engineer/parse using 3rd party tools?

Also, it's not like he can change things overnight, because there are lots
of customers/users who have repos in the old SCCS format, and he doesn't
want to completely throw away his current code just to piss off some whiny
l-k users.  At worst, if it bothers you so much, you can take up the now
seemingly forgotten Linux trait of "taking things into your own hands and
fixing it to your own needs" and write bk_evil_format_2_CVS conversion tool
instead of bitching on l-k about it.

Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://www-mddsp.enel.ucalgary.ca/People/adilger/

Re: [ANNOUNCE] BK->CVS (real time mirror)

[Brandon Low]

Well I'm a nobody in linux kernel land, but I'm a long time lurker on LKML and have been doing patchsets for the Gentoo Linux kernel for a while now.

I'd like to say that before Linus started using BK, close to 50% of the revision data that is now saved was completely lost in the process of him merging patches by hand into his repository.  I mean be realistic, do you think that Linus kept perfect track of EVERY single ' ' ';' ')' that he changed when merging a patch with minor rejects with his repo?  Do you think that every single time that he made a 1 line change or merged a 1 line change that was sent to this list it was documented and recorded?  I doubt it.  So now we are able to get a publicly available CVS repository with close to two times the data that was ever available before, and infinitely more than was ever available to anyone outside of Linus' own head.  

I personally think that Larry has done an amazing job supporting this project and it's goals, and I will give him a big "Thanks for all your support and hard work" at this time.  I think that those of you complaining about this as bitmover clearing the road to steal our data should take a long hard look at what you are really saying and consider what BK has given us that we never had before because nothing before was ever usable by Linus.

Thanks for reading (those who do),

--Brandon Low
Gentoo Linux Senior Developer

Re: [ANNOUNCE] BK->CVS (real time mirror)

[H. Peter Anvin]

Larry McVoy wrote:
>>I thought that BK has been able to export everything to a text file
>>since the first version.
> 
> 
> bk export -tpatch -r1.900 > patch.1.900
> bk changes -v -r1.900 > comments.1.900
> 
> Been there forever.  So has ways to get all the metadata from the command
> line without having to reverse engineer the file format.  See
> 
>     http://www.bitkeeper.com/manpages/bk-prs-1.html
> 
> it's all there.  Always has been.
> 
> Wayne wanted me to point that it is easy to write the BK to CVS exporter
> completely from the command line, we prototyped it that way, the only
> reason we rewrote part of it in C was for performance.  The point being
> that you guys could have done this yourself without help from us because
> all the metadata is right there.  Ditto for anyone else worried about 
> getting their data out of BK now or in the future.  The whole point of
> prs is to be able to have a will-always-work way to get at the data or
> the metadata, it makes the file format a non-issue.  
>

This is a Good Thing[TM] for a whole bunch of reasons.

Maybe this output could be made available automatically in addition to
the CVS tree?  If bandwidth is a concern then I reiterate what I said
offline yesterday, if you can give me a ballpark idea of what the
requirements seem to be I'll start hunting for a place to park a
.kernel.org server dedicated to this task.

	-hpa

Re: [ANNOUNCE] BK->CVS (real time mirror)

[Alan Cox]

On Wed, 2003-03-12 at 17:08, Roman Zippel wrote:
> this was one of the conditions for the bk usage, so Larry cannot say, that 
> you only get all the data if you use bk. If cvs can't represent all the 
> information, we have to find another solution.

CVS can't represent it all because CVS isnt up to the job. If the rest
exists as comments then its your problem to write a VCS that can extract
the comment data and represent it in full

Re: [ANNOUNCE] BK->CVS (real time mirror)

[Martin J. Bligh]

> You're missing the point. I am not against the CVS->BK gateway. I'm all
> for it. But it's kind of sour given that he now wants to change the disk
> format of the repo to make it harder to get the data from it.
> 
> If all he announced was "you now have a CVS->BK repo", I wouldn't be
> complaining, I'd be patting him on the back.

As long as we continue to get all the data in an open format, I'm
not sure this really matters, personally. If there's some data loss,
let's focus on that issue ... but it seems there isn't at the moment.

I'd rather we *didn't* go trying to clone BK and make it file-format 
compatible underneath ... that seems more incendiary than useful. 
Cloning other products is always a loosing game, the best you can do 
is catch them. Personally, I'd prefer we spent the effort making a 
usable simple SCM that 95% of us can use that does merges and stuff, 
and not bother trying to follow someone else in file format.

Of course, I'm in no position to dictate to others what they should
implement, do what you like ... just my personal opinion. But there's
always the possiblity we can make something that fits kernel development
*better*, rather than playing catchup to BK all the time ;-)

M.

Re: [ANNOUNCE] BK->CVS (real time mirror)

[Larry McVoy]

> Of course, I'm in no position to dictate to others what they should
> implement, do what you like ... just my personal opinion. But there's
> always the possiblity we can make something that fits kernel development
> *better*, rather than playing catchup to BK all the time ;-)

I like it when I agree with people, especially you since we've bumped
heads.  It's much more fun to agree...

My personal opinion is that BK maps only so so well onto the kernel
development effort.  It's not horrible, it's closer than any other SCM,
but it could be better.  The kernel guys tend to be "more loose" than
commercial guys, i.e., stuff is tried, it sits in Alan's tree for a
while or DaveJ's tree and then is rejected if it is found to be bad.
You really need a sort of "lossy" SCM system, one which is willing to
throw data away.  BK is absolutely not about losing information, we view
everything as valuable, even bad ideas.  That matches the commercial
world better than the Linux world.

I _think_ that Arch is closer.  You will definitely give up some stuff
if you move to Arch but you will also gain some stuff.  Arch is willing
to pick and choose, we aren't, we're sort of an all or nothing answer.
Pavel is all hot and bothered about PRCS but PRCS is sort of BK without
the distribution, gui tools, and scripting.  It's a step backwards as
far as I can tell (don't get me wrong, we've acknowledged the coolness
of PRCS on our website for years and I tried to team up with Josh, I'm
a fan).  You should really look at Arch, it may be a better fit.  And 
these days, if you could find a better fit, none of us at BitMover 
would shed a tear if you moved off BK.  This has *not* been a pleasant
experience for us.
-- 
---
Larry McVoy              lm at bitmover.com          http://www.bitmover.com/lm

Re: [ANNOUNCE] BK->CVS (real time mirror)

[Theodore Ts'o]

On Wed, Mar 12, 2003 at 10:03:04AM -0800, Larry McVoy wrote:
> > I thought that BK has been able to export everything to a text file
> > since the first version.
> 
> bk export -tpatch -r1.900 > patch.1.900
> bk changes -v -r1.900 > comments.1.900
> 
> Been there forever.

More importantly, even if someone isn't allowed to use the BK command
line tool because once upon time, a long time ago, they submitted a
patch to arch or subversion, they can still find someone is allowed to
set up a bk daemon under the terms of the FUL, connect to the BK
daemon using a http client, and extract the full diff of any changeset
that way.  This doesn't have to be the bkd on bkbits.net; anyone who
is authorized to use BK under the terms of the FUL can set up a bk
daemon to be listening on a port of any machine for which they have
shell access (it doesn't even require root privs).  And every last
changeset can always be made available using this path.

So to the people are complaining that they won't be able to get out
their data if a future version of BK uses a more powerful
representation than SCCS files ---- would you like some more whine
with your cheese?

						- Ted

Re: [ANNOUNCE] BK->CVS (real time mirror)

[Roman Zippel]

Hi,

On Thu, 13 Mar 2003, Theodore Ts'o wrote:

> So to the people are complaining that they won't be able to get out
> their data if a future version of BK uses a more powerful
> representation than SCCS files ---- would you like some more whine
> with your cheese?

Oh, thank you, but I'd like some of the stuff that all the bk fan boys 
must be taking.

bye, Roman

Re: [ANNOUNCE] BK->CVS (real time mirror)

[Roman Zippel]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1059 bytes --]

Hi,

On 12 Mar 2003, Alan Cox wrote:

> > this was one of the conditions for the bk usage, so Larry cannot say, that 
> > you only get all the data if you use bk. If cvs can't represent all the 
> > information, we have to find another solution.
> 
> CVS can't represent it all because CVS isnt up to the job. If the rest
> exists as comments then its your problem to write a VCS that can extract
> the comment data and represent it in full

This would require the full data, I looked at it and neither the CVS tree 
nor the Web interface has everything.
- the changeset ids are missing, if bk does its renumbering thing 
ids in comments become useless
- complete changesets are missing, RCS is quite capable of branches, so 
there is no need of merging patches into a single patch.
- merge information is missing, e.g. what branch was merged into which 
changeset.

I attached an example RCS file, how that could look like. I had to guess 
the merging information and tags are missing.
It would be really useful to have this information as well.

bye, Roman

[-- Attachment #2: Type: APPLICATION/octet-stream, Size: 60794 bytes --]

BK->CVS (2.4 + 2.5 updates)

[Larry McVoy]

Updates from the latest 2.4 and 2.5 BK trees have been applied to the
kernel.bkbits.net tree.  If you did a checkout, try an update and let
me know how it goes.
-- 
---
Larry McVoy              lm at bitmover.com          http://www.bitmover.com/lm

Re: [ANNOUNCE] BK->CVS (real time mirror)

[Andrea Arcangeli]

On Tue, Mar 11, 2003 at 08:39:19PM -0800, H. Peter Anvin wrote:
> Personally, I value my freedom to hack on whatever I want a lot more
> than the convenience of BK. [..]

Same here ;)

Andrea

Re: [ANNOUNCE] BK->CVS (real time mirror)

[Andrea Arcangeli]

On Wed, Mar 12, 2003 at 01:47:10PM -0500, Ben Collins wrote:
> On Wed, Mar 12, 2003 at 07:38:06PM +0100, Arador wrote:
> > On Tue, 11 Mar 2003 23:16:21 -0500
> > Ben Collins <bcollins@debian.org> wrote:
> > 
> > > You've made quite a marketing move. It's obvious to me, maybe not to
> > > others. By providing this CVS gateway, you make it almost pointless to
> > > work on an alternative client. Also by providing it, you make it easier
> > 
> > I don't think so. This also bits Larry. If he does well enought, there'll be
> > some people here that won't use bitkeeper just because they can use the cvs
> > gateway and they don't need/miss the features they could get with bk.
> > 
> > And i don't think it avoids creating a free bk clone. I guess that there's
> > a lot of people out there interested in such tool, and not only for kernel
> > development; this won't stop them.
> > 
> > As far as i can see; Larry is just wasting time (money) to help the kernel
> > development and people who doesn't use BK just because it isn't free. And
> > he's not charging me, so i find this a good movement for everybody. I only
> > can say thanks.
> 
> You're missing the point. I am not against the CVS->BK gateway. I'm all
> for it. But it's kind of sour given that he now wants to change the disk
> format of the repo to make it harder to get the data from it.

Ben,

You shouldn't care less of the disk format. You *can't* run bk in the
first place to reach those files, it's by pure luck that somebody is
been fine to give away his right to write free software (oh and
proprietary software too but we don't care :) in the SCM arena and to
provide this info to you via rsync or whatever proxy or open protocol
that tytso mentioned is doable.

Before you can remotely care about the disk format you've to reverse
engeneer the network protocol first, having more proprietary stuff there
won't make differences for us. And of course it makes perfect sense for
Larry to hide the stuff better, but even if he encrypts it, the secret
key has to be in the bk binary, I mean, it's all in open source assembly
anyways, if you figured out the network protcol, you shouldn't have an
order of magnitude more of troubles to figure out the new file format
too. NOTE: I don't want to discuss the legal details of reading the
open source assembly, this was only an example ;).

really, what we care is the data, and what I discussed in the last weeks
with Larry about the kernel CVS at first sigh seems enough for kernel
developers, what matters is the _mainline_ evolution.  All other trees
matters much less (and NOTE: all important non mainline trees don't use
bitkeeper anyways). If getting the changesets with dates will be too
hard I assume Larry could help on it. Some script should do it pretty
well thanks to the logical tag in the log. I know it's not the most
useful format for export but this is reliable, documented and open and
it makes it trivial to checkout and search the file logs. which makes it
very usable immediatly w/o the need of new software which is good for us
kernel developers in the short term. This is a good short/mid term solution.

IMHO cloning bitkeeper would be an option if Larry would be supporting
it, but that is obviously not the case.

There is no point to complain about the change of format of files in the
Larry has all the rights to change the file format even after you
reverse engeneered it the first second third fourth time, so all your
effort will break in seconds.  You can spend the rest of your life to
keep up with Larry and he'll always be ahead of you. We have to do this
with the SMB protocol because there's no open ""exporter"", but here
Larry provided the data, and the data belongs to the community in the
first place so there's no need to slowdown innovation here trying to
catch up with closed proprietary protocols.  And note: if you don't like
that linux is developed with bk you should speak with Linus not with
Larry.  That is Linus's choice, Larry couldn't make that change.

If you complain about the file format change, it means you realized
right now you did a mistake in depending on bk in the first place.

I think we reached a point of balance here that will solve all the
collisions. The CVS is a "stability" point. The lack of
data-availability with CVS or similar open protocol would force us to
reverse engeneer bk to access the data, and the availability of CVS
immediatly make us wasting time reverse engeneering bk.  Cloning
bitkeeper is a waste of time if the CVS just exports the data correctly.

Please focus on this: the only thing we miss is the visibility of the
jfs tree and similar other bits that aren't even guaranteed to be merged
in mainline. But that doesn't worry me at all, in one year from now if
the jfs tree didn't merge correctly it won't matter what was in such
dead tree.

If you want to contribute, stop these threads, and start importing CVS
into a more powerful SCM and let us know an URL where we can access the
data from there. I will only answer to a working URL, either that or
live with CVS. The SCM can be evolved over tiem. If this new underground
domain will be better than bitkeeper than jfs and Linus as well could
join us in the future.  In the meantime CVS will do fine and it
guarantees the openess of the linux info. As you probably know I don't
have much time in helping with SCM developement by I can t try give my
$.02 (or at the very least I want to be still allowed to give my $.02!  ;).

I won't answer further emails about this issue to avoid hurting the l-k
traffic too much (the last bk threads even made me overlook the BK->CVS
announcement after a one day and half of email backlog go figure ;).

And of course many thanks to Larry for the BK->CVS effort! While I think
it was due, it certainly takes some relevant effort to do it.

Andrea

Re: [ANNOUNCE] BK->CVS (real time mirror)

[Pavel Machek]

Hi!

> (actually Wayne Scott) did was to write a graph traversal alg which
> finds the longest path through the revision history which includes
> all tags.  For the 2.5 tree, that is currently 8298 distinct points.
> Each of those points has been captured in CVS as a commit.  If we did

As far as I can see, linux-2.5 repository has over 17000 ChangeSets,
that means half the granularity. Would it be possible to use cvs branches
to capture tree structure and have special form of commit comment
"this is merge of changeset 1.2.3.4"?
That way BK->CVS conversion could
preserve all the data...
				Pavel 
-- 
				Pavel
Written on sharp zaurus, because my Velo1 broke. If you have Velo you don't need...

Re: [ANNOUNCE] BK->CVS (real time mirror)

[Wayne Scott]

From: Pavel Machek <pavel@suse.cz>
> As far as I can see, linux-2.5 repository has over 17000 ChangeSets,
> that means half the granularity.

I assume this has already been answered since this is Monday morning
and I haven't finished my mountain of email (I try not to read it on
weekends), but I will answer this anyway.

The ChangeSet file has many csets and we only capture around 1/2 of
them in CVS ChangeSet file.  The extra ChangeSets are grouped together
with the merge cset where they were added to the path we are
recording.  That is correct, but it is not the whole story.

What happens is that most csets modifiy a non overlapping set of
files.  So while we didn't get every delta to the ChangeSet file, we
did capture >90% of the actual changes to the source files in the
tree.

Perhaps that will help explain things.

-Wayne

Re: [ANNOUNCE] BK->CVS (real time mirror)

[Pavel Machek]

Hi!

> > As far as I can see, linux-2.5 repository has over 17000 ChangeSets,
> > that means half the granularity.
> 
> I assume this has already been answered since this is Monday morning
> and I haven't finished my mountain of email (I try not to read it on
> weekends), but I will answer this anyway.
> 
> The ChangeSet file has many csets and we only capture around 1/2 of
> them in CVS ChangeSet file.  The extra ChangeSets are grouped together
> with the merge cset where they were added to the path we are
> recording.  That is correct, but it is not the whole story.
> 
> What happens is that most csets modifiy a non overlapping set of
> files.  So while we didn't get every delta to the ChangeSet file, we
> did capture >90% of the actual changes to the source files in the
> tree.

Oh, so there's extra magic.

Question, through: why is it impossible / infeasible to use CVS
branches to capture *full* information? Merge would then say
"(changeset 1.2345, merge from 1.23.4.5)" or similar...

								Pavel

-- 
Horseback riding is like software...
...vgf orggre jura vgf serr.