Re: Efficient way to import snapshots?

[Linus Torvalds <torvalds@linux-foundation.org>]

On Mon, 30 Jul 2007, Craig Boston wrote:
> > 
> > So I _seriously_ hope that about 65 of those 67 seconds was the "cvs 
> > update -d" or something like that. 
> 
> No, the only thing included in that is
> 
> git ls-files -o | git update-index --add --stdin
> git commit -a -m "${COMMITMSG}"

Ouch.

> > Anything that takes a minute in git is way way *way* too slow. Any 
> > half-way normal git operations should take less than a second.
> 
> That said, I don't think it's git's fault.  I think most of the time is
> spent calling stat() on all the files.  The machine that took 60 seconds
> isn't what I'd call top-of-the-line:
> 
> 1st or maybe 2nd-gen Willamette CPU
> 512MB memory (stupid motherboard that won't accept more)
> Slow disks in RAID-5 configuration
> Running ZFS with less than half of the recommended minimum memory, to
> the point where I had to reduce the number of vnodes that the kernel is
> allowed to cache to avoid running out of KVA

Oh, ok. Solaris.

With slow pathname lookup, and hard limits on the inode cache sizes.

Git really normally avoids reading the data, so even in 512M you should 
_easily_ be able to cache the metadata (directory and inodes), which is 
all you need. But yeah, Linux will probably do that a whole lot more 
aggressively than Solaris does.

[ And to be honest, any CVS update would probably have blown the caches on 
  Linux too - I don't know what all CVS ends up doing, but from past
  experience, I'll bet it's not good ]

But just for comparison, on a lowly 480MB mac-mini (running Linux, not OS 
X, of course - and the 480MB is because the graphics is UMA and takes 
part of the 512MB total), and the kernel archive (which is just 22k files, 
not 37k), with a laptop drive:

 - cold-cache "git status":
	real    0m17.975s
	user    0m1.098s
	sys     0m0.539s

 - rerunning it immediately afterwards:
	real    0m1.079s
	user    0m0.896s
	sys     0m0.183s

so the target really _should_ generally be one second.

But yeah, in order to hit that target, you definitely do want to keep the 
metadata cached, and I guess that means more than 512M on Solaris.

			Linus