Re: Efficient way to import snapshots?

[Craig Boston <craig@olyun.gank.org>]

On Mon, Jul 30, 2007 at 04:30:22PM -0700, Linus Torvalds wrote:
> > # On branch cvs_RELENG_4
> > nothing to commit (working directory clean)
> > git: 67.65 seconds
> 
> So I _seriously_ hope that about 65 of those 67 seconds was the "cvs 
> update -d" or something like that. 

No, the only thing included in that is

git ls-files -o | git update-index --add --stdin
git commit -a -m "${COMMITMSG}"

> Anything that takes a minute in git is way way *way* too slow. Any 
> half-way normal git operations should take less than a second.

That said, I don't think it's git's fault.  I think most of the time is
spent calling stat() on all the files.  The machine that took 60 seconds
isn't what I'd call top-of-the-line:

1st or maybe 2nd-gen Willamette CPU
512MB memory (stupid motherboard that won't accept more)
Slow disks in RAID-5 configuration
Running ZFS with less than half of the recommended minimum memory, to
the point where I had to reduce the number of vnodes that the kernel is
allowed to cache to avoid running out of KVA

A simple find(1) over the CVS checkout directory takes almost as long.
I don't think it has enough memory to cache the whole thing.  Actually I
know it can't since maxvnodes is set to 25,000 and there's 37,000 files
in the cvs checkout, so it will have to pull some directory entries from
disk regardless.

Just to be sure, I copied the cvs checkout directory and git repository
to a newer, faster dual-core machine with plenty of memory available for
caching.

The first run of 'git status' (cold cache):
git status  1.08s user 3.68s system 13% cpu 34.043 total

The second run:
git status  1.05s user 2.68s system 85% cpu 4.373 total

Based on that I'm fairly confident that most of the 60 seconds is being
spent waiting on data from the disks.  On a tmpfs filesystem I can get
it even faster (1.897 seconds)

As it's a file server for which network is the usual bottleneck, and all
the git operations will be running out of cron, I'm not too worried
about it.

Craig