A simple request (was Re: boring BK stats)

A simple request (was Re: boring BK stats)

[Walter Landry]

Jeff Garzik wrote:
> If you can't fit a whole tree including metadata into RAM, though,
> BK crawls... Going from "bk citool" at the command line to actually
> seeing the citool window approaches five minutes of runtime, on this
> 200MB laptop... [my dual athlon with 512MB RAM corroborates your
> numbers, though] "bk -r co -Sq" takes a similar amount of time...
> 
> I also find that BK brings out the worst in the 2.4 kernel
> elevator/VM... mouse clicks in Mozilla take upwards of 10 seconds to
> respond, when "bk -r co -Sq" is running on this laptop [any other
> read-from-disk process behaves similarly]. And running any two BK
> jobs at the same time is a huge mistake. Two "bk -r co -Sq" runs
> easily take four or more times longer than a single run. Ditto for
> consistency checks, or any other disk-intensive activity BK indulges
> in.

Hello,

What kind of CPU and hard drive do your two machines above have?  I'm
a developer for arch[1], and I'm wondering how fast things can get.

Note: If you answer, you'll certainly be aiding arch development.  It
      might be interpreted as "develop[ing] ... a product which
      contains substantially similar capabilities of the BitKeeper
      Software, or, in the reasonable opinion of BitMover, competes
      with the BitKeeper Software".  So you might lose the ability to
      use the free license.  But I'll let you decide if you want to
      help us.

Thank you,
Walter Landry
wlandry@ucsd.edu

[1] www.fifthvision.net/Arch

Re: A simple request (was Re: boring BK stats)

[Jeff Garzik]

Walter Landry wrote:
> Jeff Garzik wrote:
> 
>>If you can't fit a whole tree including metadata into RAM, though,
>>BK crawls... Going from "bk citool" at the command line to actually
>>seeing the citool window approaches five minutes of runtime, on this
>>200MB laptop... [my dual athlon with 512MB RAM corroborates your
>>numbers, though] "bk -r co -Sq" takes a similar amount of time...
>>
>>I also find that BK brings out the worst in the 2.4 kernel
>>elevator/VM... mouse clicks in Mozilla take upwards of 10 seconds to
>>respond, when "bk -r co -Sq" is running on this laptop [any other
>>read-from-disk process behaves similarly]. And running any two BK
>>jobs at the same time is a huge mistake. Two "bk -r co -Sq" runs
>>easily take four or more times longer than a single run. Ditto for
>>consistency checks, or any other disk-intensive activity BK indulges
>>in.
> 
> 
> Hello,
> 
> What kind of CPU and hard drive do your two machines above have?  I'm
> a developer for arch[1], and I'm wondering how fast things can get.

The laptop has 200MB RAM, and mozilla and a ton of xterms loaded.  IDE 
drives w/ Intel PIIX4 controller.  The Dual Athlon has 512MB RAM, and I 
forget what kind of IDE controller -- I think AMD.  IDE drives as well.

BitKeeper must scan the entire tree when doing a checkin or checkout, so 
that is impossible to optimize at the SCM level without compromising 
features...  if your source tree takes up ~190MB on disk, you have 200MB 
of RAM total, and you need to sequentially scan the entire thing, there 
is nothing that can be done at either the OS or app level... You're just 
screwed.  Things are extremely fast on the Dual Athlon because the 
entire tree is in RAM.


> Note: If you answer, you'll certainly be aiding arch development.  It
>       might be interpreted as "develop[ing] ... a product which
>       contains substantially similar capabilities of the BitKeeper
>       Software, or, in the reasonable opinion of BitMover, competes
>       with the BitKeeper Software".  So you might lose the ability to
>       use the free license.  But I'll let you decide if you want to
>       help us.


Don't be silly:  I am more than willing to help, via answering 
questions.  If I had time I would help code.  I'm not religious about 
"only BitKeeper" to the exclusion of helping open source projects.  I 
simply use what is most expedient and productive for my kernel 
development workflow.

	Jeff

Re: A simple request (was Re: boring BK stats)

[Larry McVoy]

> The laptop has 200MB RAM, and mozilla and a ton of xterms loaded.  IDE 
> drives w/ Intel PIIX4 controller.  The Dual Athlon has 512MB RAM, and I 
> forget what kind of IDE controller -- I think AMD.  IDE drives as well.
> 
> BitKeeper must scan the entire tree when doing a checkin or checkout, so 
> that is impossible to optimize at the SCM level without compromising 
> features...  if your source tree takes up ~190MB on disk, you have 200MB 
> of RAM total, and you need to sequentially scan the entire thing, there 
> is nothing that can be done at either the OS or app level... You're just 
> screwed.  Things are extremely fast on the Dual Athlon because the 
> entire tree is in RAM.

In low memory situations you really want to run the tree compressed.  
ON a fast machine do a "bk -r admin -Z" and then clone that onto your
laptop.  I think that will drop the tree to about 145MB which will
help, maybe.  I suspect that you use enough of the rest of your 200MB
that it still won't fit.

For the checkouts, always do a "bk -r get -S" the -S doesn't check out the
file again if it is already there.  We could make that the default but
it is an interface change.  A fairly minor one though.

We've got some other fixes in the pipeline for the checkin and integrity
check pass.

There is only so much we can do when you are trying to cram 10 pounds of
crap in a 5 pound bag :(
-- 
---
Larry McVoy            	 lm at bitmover.com           http://www.bitmover.com/lm 

Re: A simple request (was Re: boring BK stats)

[Jeff Garzik]

Larry McVoy wrote:
>>The laptop has 200MB RAM, and mozilla and a ton of xterms loaded.  IDE 
>>drives w/ Intel PIIX4 controller.  The Dual Athlon has 512MB RAM, and I 
>>forget what kind of IDE controller -- I think AMD.  IDE drives as well.
>>
>>BitKeeper must scan the entire tree when doing a checkin or checkout, so 
>>that is impossible to optimize at the SCM level without compromising 
>>features...  if your source tree takes up ~190MB on disk, you have 200MB 
>>of RAM total, and you need to sequentially scan the entire thing, there 
>>is nothing that can be done at either the OS or app level... You're just 
>>screwed.  Things are extremely fast on the Dual Athlon because the 
>>entire tree is in RAM.
> 
> 
> In low memory situations you really want to run the tree compressed.  
> ON a fast machine do a "bk -r admin -Z" and then clone that onto your
> laptop.  I think that will drop the tree to about 145MB which will
> help, maybe.  I suspect that you use enough of the rest of your 200MB
> that it still won't fit.

Yeah, I don't think that will help at all, given that X and KDE and all 
its acoutrements are loaded...  I would rather run uncompressed anyway :)


> For the checkouts, always do a "bk -r get -S" the -S doesn't check out the
> file again if it is already there.  We could make that the default but
> it is an interface change.  A fairly minor one though.

I do "bk -r co -Sq", is the above faster than that?


> We've got some other fixes in the pipeline for the checkin and integrity
> check pass.
> 
> There is only so much we can do when you are trying to cram 10 pounds of
> crap in a 5 pound bag :(

indeed :)  That's why I keep repeating that it's not BK's fault, and 
keep pointing out that my Dual Athlon with plenty of RAM does multiple 
simultaneous checks/checkins quite rapidly.

	Jeff

Re: A simple request (was Re: boring BK stats)

[Theodore Ts'o]

On Thu, Oct 10, 2002 at 07:28:18AM -0700, Larry McVoy wrote:
> 
> In low memory situations you really want to run the tree compressed.  
> ON a fast machine do a "bk -r admin -Z" and then clone that onto your
> laptop.  I think that will drop the tree to about 145MB which will
> help, maybe.  I suspect that you use enough of the rest of your 200MB
> that it still won't fit.

I believe it's the case that all of the trees on bkbits.net are
compressed, so if you originally cloned your kernel tree off of
bkbits.net, it should already be compressed.  Larry, would this be a
correct statement?

						- Ted

Re: A simple request (was Re: boring BK stats)

[Rogier Wolff]

On Thu, Oct 10, 2002 at 07:28:18AM -0700, Larry McVoy wrote:
> > The laptop has 200MB RAM, and mozilla and a ton of xterms loaded.  IDE 
> > drives w/ Intel PIIX4 controller.  The Dual Athlon has 512MB RAM, and I 
> > forget what kind of IDE controller -- I think AMD.  IDE drives as well.
> > 
> > BitKeeper must scan the entire tree when doing a checkin or checkout, so 

[...]

> In low memory situations you really want to run the tree compressed.  
> ON a fast machine do a "bk -r admin -Z" and then clone that onto your
> laptop.  I think that will drop the tree to about 145MB which will
> help, maybe.  I suspect that you use enough of the rest of your 200MB
> that it still won't fit.

[...]
> There is only so much we can do when you are trying to cram 10 pounds of
> crap in a 5 pound bag :(

The reason that one or two years ago my "diff+multiple trees" beat
bitkeeper on the performance front was that diff would only touch
inode-metadata, and not the files themselves. You can cache the
file-metadata (inodes) of a 200M tree in a couple of megabytes of RAM.

			Roger. 

-- 
** R.E.Wolff@BitWizard.nl ** http://www.BitWizard.nl/ ** +31-15-2600998 **
*-- BitWizard writes Linux device drivers for any device you may have! --*
* The Worlds Ecosystem is a stable system. Stable systems may experience *
* excursions from the stable situation. We are currenly in such an       * 
* excursion: The stable situation does not include humans. ***************

Re: A simple request (was Re: boring BK stats)

[Rogier Wolff]

On Fri, Oct 11, 2002 at 03:35:38PM +0200, Rogier Wolff wrote:
> On Thu, Oct 10, 2002 at 07:28:18AM -0700, Larry McVoy wrote:
> > > The laptop has 200MB RAM, and mozilla and a ton of xterms loaded.  IDE 
> > > drives w/ Intel PIIX4 controller.  The Dual Athlon has 512MB RAM, and I 
> > > forget what kind of IDE controller -- I think AMD.  IDE drives as well.
> > > 
> > > BitKeeper must scan the entire tree when doing a checkin or checkout, so 
> 
> [...]
> 
> > In low memory situations you really want to run the tree compressed.  
> > ON a fast machine do a "bk -r admin -Z" and then clone that onto your
> > laptop.  I think that will drop the tree to about 145MB which will
> > help, maybe.  I suspect that you use enough of the rest of your 200MB
> > that it still won't fit.
> 
> [...]
> > There is only so much we can do when you are trying to cram 10 pounds of
> > crap in a 5 pound bag :(
> 
> The reason that one or two years ago my "diff+multiple trees" beat
> bitkeeper on the performance front was that diff would only touch
> inode-metadata, and not the files themselves. You can cache the
> file-metadata (inodes) of a 200M tree in a couple of megabytes of RAM.

Jeff: 

The trick is that I would have hardlinked trees. Thus 

	linux-2.2.18.clean/drivers/char/tty_ioctl.c
and
	linux-2.2.18.rio/drivers/char/tty_ioctl.c

would have the same inode number, and diff wouldn't even bother to
open the file.... 

				Roger. 

-- 
** R.E.Wolff@BitWizard.nl ** http://www.BitWizard.nl/ ** +31-15-2600998 **
*-- BitWizard writes Linux device drivers for any device you may have! --*
* The Worlds Ecosystem is a stable system. Stable systems may experience *
* excursions from the stable situation. We are currenly in such an       * 
* excursion: The stable situation does not include humans. ***************

Re: A simple request (was Re: boring BK stats)

[Rogier Wolff]

On Fri, Oct 11, 2002 at 04:08:28PM +0200, Rogier Wolff wrote:
> Jeff: 
> 
> The trick is that I would have hardlinked trees. Thus 
> 
> 	linux-2.2.18.clean/drivers/char/tty_ioctl.c
> and
> 	linux-2.2.18.rio/drivers/char/tty_ioctl.c
> 
> would have the same inode number, and diff wouldn't even bother to
> open the file.... 

FYI: 

Just tested the how long it takes to make a diff between my
2.2.18-clean tree and the 2.2.18.rio tree:

cold cache over NFS:     25 seconds
warm cache over NFS:     13 seconds
warm cache LOCAL disk:    0.34 seconds. 

					Roger. 

-- 
** R.E.Wolff@BitWizard.nl ** http://www.BitWizard.nl/ ** +31-15-2600998 **
*-- BitWizard writes Linux device drivers for any device you may have! --*
* The Worlds Ecosystem is a stable system. Stable systems may experience *
* excursions from the stable situation. We are currenly in such an       * 
* excursion: The stable situation does not include humans. ***************

Re: A simple request

[Walter Landry]

Jeff Garzik <jgarzik@pobox.com> wrote:
> Walter Landry wrote:
> > Jeff Garzik wrote:
> > 
> >>If you can't fit a whole tree including metadata into RAM, though,
> >>BK crawls... Going from "bk citool" at the command line to actually
> >>seeing the citool window approaches five minutes of runtime, on this
> >>200MB laptop... [my dual athlon with 512MB RAM corroborates your
> >>numbers, though] "bk -r co -Sq" takes a similar amount of time...
> >>
> >>I also find that BK brings out the worst in the 2.4 kernel
> >>elevator/VM... mouse clicks in Mozilla take upwards of 10 seconds to
> >>respond, when "bk -r co -Sq" is running on this laptop [any other
> >>read-from-disk process behaves similarly]. And running any two BK
> >>jobs at the same time is a huge mistake. Two "bk -r co -Sq" runs
> >>easily take four or more times longer than a single run. Ditto for
> >>consistency checks, or any other disk-intensive activity BK indulges
> >>in.
> > 
> > 
> > Hello,
> > 
> > What kind of CPU and hard drive do your two machines above have?  I'm
> > a developer for arch[1], and I'm wondering how fast things can get.
> 
> The laptop has 200MB RAM, and mozilla and a ton of xterms loaded.  IDE 
> drives w/ Intel PIIX4 controller.  The Dual Athlon has 512MB RAM, and I 
> forget what kind of IDE controller -- I think AMD.  IDE drives as well.

Sorry, I wasn't specific enough.  What is the clock speed (MHz) of the
CPU and the RPM and/or results of hparm -t of the hard drive?

> BitKeeper must scan the entire tree when doing a checkin or checkout, so 
> that is impossible to optimize at the SCM level without compromising 
> features...  if your source tree takes up ~190MB on disk, you have 200MB 
> of RAM total, and you need to sequentially scan the entire thing, there 
> is nothing that can be done at either the OS or app level... You're just 
> screwed.  Things are extremely fast on the Dual Athlon because the 
> entire tree is in RAM.

How fast?  If you have, for example, the 2.4.18 tree and apply the
2.4.19 patches, approximately how long does it take for citool to show
up, showing all modified files (less than a second? 10 seconds? a
minute?)

> > Note: If you answer, you'll certainly be aiding arch development.  It
> >       might be interpreted as "develop[ing] ... a product which
> >       contains substantially similar capabilities of the BitKeeper
> >       Software, or, in the reasonable opinion of BitMover, competes
> >       with the BitKeeper Software".  So you might lose the ability to
> >       use the free license.  But I'll let you decide if you want to
> >       help us.
> 
> 
> Don't be silly:  I am more than willing to help, via answering 
> questions.

Thanks,
Walter Landry
wlandry@ucsd.edu