RE: kernel ftp ?

RE: kernel ftp ?

[Zehetbauer Thomas]

BitKeeper is commercial software and therefore not very like to succeed
in the path of kernel devlopment. I think we've had this discussion a
thousand times on lkml. The reasons are numerous even if BitKeeper is
licensed for free to open source developers.

Firstly it is unlikely to be distributed with your favourite linux
distribution. You have to find and download the software first. Of
course I agree that this is a problem of every third party software.

As it is unlikely that a package or at least a binary for your favourite
linux distribution is available it is a maintenance nightmare to track
down library problems and do release upgrades or uninstall it.

To start working with the software you will need to learn another set of
commands, options and parameters.

If are working on some not to be public changes you will have to buy
BitKeeper or use a second source code control system.

And finally if I encounter a problem at 3am in the morning there is no
sourcecode to track it down, I will have to wait until the next day to
call and pay for support.

These are the main reasons I would definitely prefer CVS/FTP/rsync
access to the kernel tree!

Tom

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: kernel ftp ?

[Larry McVoy]

On Mon, Jul 23, 2001 at 11:44:02AM +0200, Zehetbauer Thomas wrote:
> [BK flame]

Thomas, sorry you feel that way.  List, if you have any questions about
our take on Thomas' statements, some of which we feel are inaccurate,
contact me privately.
--
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: kernel ftp ?

[Troy Benjegerdes]

On Mon, Jul 23, 2001 at 08:38:10AM -0700, Larry McVoy wrote:
>
> On Mon, Jul 23, 2001 at 11:44:02AM +0200, Zehetbauer Thomas wrote:
> > [BK flame]
>
> Thomas, sorry you feel that way.  List, if you have any questions about
> our take on Thomas' statements, some of which we feel are inaccurate,
> contact me privately.

Thomas does have some valid points.

Larry, I also understand why you can't release Bitkeeper as 'free
software'.  Please understand that we, as a community of free software
developers can NOT become dependent on any non-free software.

Most of us doing development are using BK because it works better.
However, Not providing access to the source via rsync or some other less
efficient protocol with a 'free software' implementation would be more
than a bit hypocritical.

Let's please end this discussion and any bitkeeper advocacy/disadvocacy.

(I think most people would be happy to discuss this again if either BK is
released under a license that meets the debian free software guidelines,
or another 'free' implementation is developed that allows pulls and
clones.)

--
Troy Benjegerdes | master of mispeeling | 'da hozer' |  hozer@drgw.net
-----"If this message isn't misspelled, I didn't write it" -- Me -----
"Why do musicians compose symphonies and poets write poems? They do it
because life wouldn't have any meaning for them if they didn't. That's
why I draw cartoons. It's my life." -- Charles Shulz

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

kernel ftp ?

[Giuliano Pochini]

Is there an ftp server with the latest ppc kernels or I have to use bk ?

Bye.


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: kernel ftp ?

[Steven Hanley]

On Sun, Jul 15, 2001 at 04:10:16PM +0200, Giuliano Pochini wrote:
>
> Is there an ftp server with the latest ppc kernels or I have to use bk ?

all the latet kerneles for ppc tend to also be available with rsync, install
rsync and grab the kernels with something like

benh rsync -avz --delete-after penguinppc.org::linux-2.4-benh .

paulus rsync -avz --delete-after penguinppc.org::linux-pmac-{stable,devel,etc} .

bk rsync -avz --delete-after bitkeeper.fsmlabs.com::linuxppc_2_4 linuxppc_2_4

no idea if the bitkeeper one is stil there but hey. I use the benh kernels
generally as they work best with my pismo and work fine with my 7220/200
anyway.

        See You
            Steve

--
sjh@wibble.net http://wibble.net/~sjh/
Look Up In The Sky
   Is it a bird?  No
      Is it a plane?  No
         Is it a small blue banana?
YES

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: kernel ftp ?

[Benjamin Herrenschmidt]

>all the latet kerneles for ppc tend to also be available with rsync, install
>rsync and grab the kernels with something like
>
>benh rsync -avz --delete-after penguinppc.org::linux-2.4-benh .
>
>paulus rsync -avz --delete-after penguinppc.org::linux-pmac-
>{stable,devel,etc} .

Paul no longer maintains his own tree, his work is directly merged
in either bk _2_4 or _2_4_devel now.

>bk rsync -avz --delete-after bitkeeper.fsmlabs.com::linuxppc_2_4 linuxppc_2_4

The bk has moved, I beleive new access infos can be found at
www.penguinppc.org

Ben.


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: kernel ftp ?

[Michael Schmitz]

> Paul no longer maintains his own tree, his work is directly merged
> in either bk _2_4 or _2_4_devel now.

Which are hosted by montavista:
rsync -avz --delete source.mvista.com::linuxppc_2_4 <directory>

for the 'stable' 2.4 tree.

	Michael


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: kernel ftp ?

[Benjamin Herrenschmidt]

>
>> Paul no longer maintains his own tree, his work is directly merged
>> in either bk _2_4 or _2_4_devel now.
>
>Which are hosted by montavista:
>rsync -avz --delete source.mvista.com::linuxppc_2_4 <directory>
>
>for the 'stable' 2.4 tree.

Those are mirrors, the main repository is hosted by bitmover
(ppc.bitkeeper.com). But those mirrors should work fine as well..

Ben.


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: kernel ftp ?

[Michael Schmitz]

> >Which are hosted by montavista:
> >rsync -avz --delete source.mvista.com::linuxppc_2_4 <directory>
> >
> >for the 'stable' 2.4 tree.
>
> Those are mirrors, the main repository is hosted by bitmover
> (ppc.bitkeeper.com). But those mirrors should work fine as well..

Yep, but does the bitmover sire offer rsync access? That's what most of us
still use ...

	Michael


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: kernel ftp ?

[Larry McVoy]

On Mon, Jul 16, 2001 at 08:01:35PM +0200, Michael Schmitz wrote:
> > >Which are hosted by montavista:
> > >rsync -avz --delete source.mvista.com::linuxppc_2_4 <directory>
> > >
> > >for the 'stable' 2.4 tree.
> >
> > Those are mirrors, the main repository is hosted by bitmover
> > (ppc.bitkeeper.com). But those mirrors should work fine as well..
>
> Yep, but does the bitmover sire offer rsync access? That's what most of us
> still use ...

No, we do not and will not offer rsync or FTP access.  Those are very
bandwidth intensive services, BK uses a tiny fraction of what they use,
and you can accomplish the same thing with a

	rm -rf /tmp/exported_tree
	bk pull
	bk export -tplain /tmp/exported_tree

That's way, way, way less bytes moved to get you a perfect mirror.
I really don't care if you use BK or not to do development, that is up
to you.  But you should use it to conserve bandwidth and you must use
it if you want the data from us.

I can easily understand you not wanting to learn a new tool, or having
some other reason, valid or otherwise, not to use BK.  That's fine.
But you need to understand that anyone providing a hosting service is
spending money to do so.  We've spent about $25K to date.  Right now,
we're behind a pair of T1 speed DSL lines that cost us about $800/month.
If we fill up those lines the DSL people will shut us down and we'll
have to move to real T1 lines.  That would at least triple our costs.
I think it's a lot more than that, last I checked a fractional T1,
around 400Kbits/sec, was $1000/month.

Our way around this problem is to get you to do one "bk clone" and only
"bk pulls" after that.  That will transfer _only_ the data which has to
be transferred, nothing else.  Even that is a substantial amount when
you multiply it all out by the number of people.  We'll deal with that,
we won't deal with full copies.

I know Mvista offers rsync/ftp access and they're welcome to do so.  They
make (some) money off of Linux/PPC so they can justify it.  I suspect,
however, when they find out that FTP/rsync is filling up a T1 line and they
have to buy more bandwidth, their management may raise a stink.  Money is
money, it's one thing to spend it when there is no other choice, it's quite
another for people to be wasteful.
--
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: kernel ftp ?

[Michael Schmitz]

> > Yep, but does the bitmover sire offer rsync access? That's what most of us
> > still use ...
>
> No, we do not and will not offer rsync or FTP access.  Those are very
> bandwidth intensive services, BK uses a tiny fraction of what they use,
> and you can accomplish the same thing with a
>
> 	rm -rf /tmp/exported_tree
> 	bk pull
> 	bk export -tplain /tmp/exported_tree
>
> That's way, way, way less bytes moved to get you a perfect mirror.

Thanks. I wasn't pushing for bitmover to set up rsync, I honestly didn't
remember.

> I can easily understand you not wanting to learn a new tool, or having
> some other reason, valid or otherwise, not to use BK.  That's fine.

Thanks for your understanding - in my case it's just inertia at work. Plus
there doesn't seem to be a Debian or RPM package I could find. I don't
follow kernel development closely, I don't need to commit patches, even a
source tarball snapshot posted to some big FTP archive would suit me fine.

> Our way around this problem is to get you to do one "bk clone" and only
> "bk pulls" after that.  That will transfer _only_ the data which has to
> be transferred, nothing else.  Even that is a substantial amount when
> you multiply it all out by the number of people.  We'll deal with that,
> we won't deal with full copies.

Color me naive but wouldn't a second tier of bk or other sites alleviate
that? Provided they won't allow commits so syncing the repositories won't
get to be a headache? Mvista runs a bk mirror, incidentially. Other sites
willing to set one up might be found if necessary (what's the average disk
space requirement? You covered the bandwith aspect nicely...). Together
with a note on penguinppc.org and other pages advertising the kernel
source, to please use mirrors where possible?

	Michael


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: kernel ftp ?

[Larry McVoy]

On Mon, Jul 16, 2001 at 08:38:17PM +0200, Michael Schmitz wrote:
> Thanks for your understanding - in my case it's just inertia at work. Plus
> there doesn't seem to be a Debian or RPM package I could find. I don't
> follow kernel development closely, I don't need to commit patches, even a
> source tarball snapshot posted to some big FTP archive would suit me fine.

http://www.bitmover.com/download

There is no RPM but the binaries all go in one place.  Complain loadly enough
and we'll make RPMs.

> > Our way around this problem is to get you to do one "bk clone" and only
> > "bk pulls" after that.  That will transfer _only_ the data which has to
> > be transferred, nothing else.  Even that is a substantial amount when
> > you multiply it all out by the number of people.  We'll deal with that,
> > we won't deal with full copies.
>
> Color me naive but wouldn't a second tier of bk or other sites alleviate
> that? Provided they won't allow commits so syncing the repositories won't
> get to be a headache?

Sure that would work fine but it still means you have to install BK to
get the data.  But if you do that, yes, it makes tons of sense to have
a pile of hosts around the world providing mirrors.  And we can set it
up such that we auto-push to them from here when new stuff comes in.

I think the point I'm trying to make is that _nobody_ can afford to
provide infinite bandwidth for free.  I don't support the Mvista choice
of doing so one little bit, I think it is self destructive.  Even if
they are making money from Linux/PPC, why throw it away needlessly?
Last year there was so much money floating around the valley that noone
worried about a few grand a month.  This year people are being laid off
right and left, partially because of wasteful decisions.  I'm from the
MidWest of the US, where people are well known for "waste not, want not".
I don't see why bandwidth shouldn't fall under that as well.  And BK rocks
as a mirroring service, it's amazingly good.  One of our developers is
behind a modem.  BK works great (he hates life because surfing the net
sucks, but the BK part is fine).

I think 5 years from now you'll see people using BK, or things like it, for
doing mirroring all over the world.  It works.
--
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: kernel ftp ?

[Troy Benjegerdes]

> I think the point I'm trying to make is that _nobody_ can afford to
> provide infinite bandwidth for free.  I don't support the Mvista choice
> of doing so one little bit, I think it is self destructive.  Even if
> they are making money from Linux/PPC, why throw it away needlessly?

I don't think mvista is throwing away money needlessly.. if you notice, we
have rsync mirrors, but NOT ftp... rsync only sucks bandwidth if people
are grabbing the tree for the first time, just like bk clone does. In
fact, BK takes MORE bandwidth than rsync on a 'clone' operation because it
has to ship the complete revision history along.

> Last year there was so much money floating around the valley that noone
> worried about a few grand a month.  This year people are being laid off
> right and left, partially because of wasteful decisions.  I'm from the
> MidWest of the US, where people are well known for "waste not, want not".
> I don't see why bandwidth shouldn't fall under that as well.  And BK rocks
> as a mirroring service, it's amazingly good.  One of our developers is
> behind a modem.  BK works great (he hates life because surfing the net
> sucks, but the BK part is fine).
>
> I think 5 years from now you'll see people using BK, or things like it, for
> doing mirroring all over the world.  It works.

I think rsync has beaten you to the punch... it's already used to mirror
most of the major source repository out there, and it doesn't care if the
data is source code, tarballs, pictures, or whatnot. It also only
transfers data that has changed, like bk. I will admit that BK is finer
grained that rsync and transfers less uneeded stuff, but they are both
still on the same order of magnitude.

Out of curiosity, I tried doing a 'bk pull' of 266 changesets, and checked
how much bandwidth it took as well as how much bandwidth rsync took to
update the corresponding 'exported' source tree. (I used the stats from
'ifconfig' to figure out how much bandwidth.. this was on an otherwise
unused system)

RX bytes:135187230 (128.9 Mb)  TX bytes:5130187 (4.8 Mb)
cd linuxppc_2_4_devel; bk pull
RX bytes:138261336 (131.8 Mb)  TX bytes:5805084 (5.5 Mb)


RX bytes:140682043 (134.1 Mb)  TX bytes:8804424 (8.3 Mb)
rsync -avz --rsh=ssh --delete narn:/scratch/linuxppc_2_4_devel-rsync linuxppc_2_4_devel-rsync

wrote 1107822 bytes  read 4288794 bytes  38138.63 bytes/sec
total size is 113771809  speedup is 21.08
RX bytes:145793519 (139.0 Mb)  TX bytes:10484712 (9.9 Mb)


BK took about 3 MB, and rsync took abut 5 MB. So yes, BK is more
efficient, but the extra rsync traffic is not going to cause mvista any
great deal of trouble.


--
Troy Benjegerdes | master of mispeeling | 'da hozer' |  hozer@drgw.net
-----"If this message isn't misspelled, I didn't write it" -- Me -----
"Why do musicians compose symphonies and poets write poems? They do it
because life wouldn't have any meaning for them if they didn't. That's
why I draw cartoons. It's my life." -- Charles Shulz

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: kernel ftp ?

[Larry McVoy]

> fact, BK takes MORE bandwidth than rsync on a 'clone' operation because it
> has to ship the complete revision history along.

Wow.  Amazing insight, that.  You could say "copying 100MB takes MORE
bandwidth than copying 50MB because you have to ship the second 50MB",
and that would be an equally amazing insight.

> I think rsync has beaten you to the punch... it's already used to mirror
> most of the major source repository out there, and it doesn't care if the
> data is source code, tarballs, pictures, or whatnot. It also only
> transfers data that has changed, like bk. I will admit that BK is finer
> grained that rsync and transfers less uneeded stuff, but they are both
> still on the same order of magnitude.

That may be true for small sites, but it doesn't scale.  People tend to
update automatically, i.e., out of cron.  A null pull of a BK tree will
transfer about 9KB for the whole operation and will stat/open less than
ten files.  Rsync will stat *every* file.  In other words, rsync places
a load on your server proportional to the number of files, not number of
repositories.

The number of repositories that you could host with BK is orders of
magnitudes higher than the number you could host with rsync, holding
the bandwidth/disks/memory/CPU constant.

It's not rsync's "fault", rsync has no mechanism to know what changed
other than looking at everything, BK records that once at commit time
and then it just knows.  Rsync is great at what it does, but that
doesn't mean that what it does is the only thing that needs to be done
nor is it the best way to do it.

The fact that you can host a handful of trees on a machine with lots of
CPU and memory is rather unimpressive.  Multiply that by 10,000 and get
back to me.

If I sound sarcastic, good, I intend to be.  People being wasteful is
distasteful to me.  We live in a world of finite resources and people
constantly think there will just be more.  More money, more electricity,
more bandwidth.  That stuff is not free, someone is paying for it.

I get the feeling that you, Troy, are just arguing to win the argument
because you want to win.  That's called winning the battle and losing
the war.  Not widely considered to be a smart approach.  How about you
use your smarts to waste less instead of winning meaningless battles?
--
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: kernel ftp ?

[Lars Magne Ingebrigtsen]

Larry McVoy <lm@bitmover.com> writes:

> If I sound sarcastic, good, I intend to be.  People being wasteful is
> distasteful to me.

And people trying to push their product by inventing problems is
distasteful to me.

Get over it.  rsync won.

--
Lars Magne Ingebrigtsen

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: kernel ftp ?

[Geert Uytterhoeven]

On Sun, 22 Jul 2001, Larry McVoy wrote:
> > fact, BK takes MORE bandwidth than rsync on a 'clone' operation because it
> > has to ship the complete revision history along.
>
> Wow.  Amazing insight, that.  You could say "copying 100MB takes MORE
> bandwidth than copying 50MB because you have to ship the second 50MB",
> and that would be an equally amazing insight.

Yes, indeed we're comparing apples to oranges (I think that's how you say it in
English, in Dutch we compare apples to lemons :-)

We developers do want the extra 50 MB, but mere mortals don't. They just want a
copy of the latest version of the tree. Hacking is beyond their capabilities.

IMHO, bk en rsync both have their uses.

Gr{oetje,eeting}s,

						Geert

P.S. I do agree more or less to the other arguments in your mail, about null
     pulls and the scarcity of resources.
--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
							    -- Linus Torvalds


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: kernel ftp ?

[Timothy A. Seufert]

At 9:07 AM +0200 7/23/01, Geert Uytterhoeven wrote:
>On Sun, 22 Jul 2001, Larry McVoy wrote:
>>  > fact, BK takes MORE bandwidth than rsync on a 'clone' operation because it
>>  > has to ship the complete revision history along.
>>
>>  Wow.  Amazing insight, that.  You could say "copying 100MB takes MORE
>>  bandwidth than copying 50MB because you have to ship the second 50MB",
>>  and that would be an equally amazing insight.
>
>Yes, indeed we're comparing apples to oranges (I think that's how
>you say it in
>English, in Dutch we compare apples to lemons :-)
>
>We developers do want the extra 50 MB, but mere mortals don't. They
>just want a
>copy of the latest version of the tree. Hacking is beyond their capabilities.

Larry, would it be even remotely feasible to add a feature to bk that
checks out just the current state of the tree with no revision
history?  Or do repositories have to be identical up to the branching
point of the last mutual push/pull?  Maybe it could be done by
creating a special kind of repository that can only pull things from
the parent?

I'm thinking something along the lines of the client having no SCCS
files, sort of like doing a bk export from the server repository to
the client's machine.  Unlike export there would also be tag
information for future reference when the server needs to figure out
what's new for the client.  You could call the client side command
"bk rsync".  :)

I'm way out of my depth here so I have no idea if this is possible or
how much work it would entail.
--
Tim Seufert

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/