Re: kde.git is now online

[Nicolas Pitre <nico@cam.org>]

On Thu, 5 Apr 2007, Linus Torvalds wrote:

> Without "--full", it doesn't actually really do anything much, since it 
> will basically ignore objects that are in the pack.
> 
> With --full, there are certainly things that we could improve upon. We 
> currently tend to walk things a few times for pack contents: 
>  - first we do the SHA1 of the full pack
>  - then we go back, and unpack and fsck each entry in the pack.
> 
> So if the pack-file is too big to fit in memory, we'll basically always 
> read it at least twice (and that's ignoring the fact that delta lookup 
> will obviously seek back and forth, which makes access patterns worse).
> 
> On the other hand, there's a perfectly good reason why we don't actually 
> fsck pack-files by default. They're "stable storage". You don't normally 
> need to. So I'd not worry too much about fsck performance.

Well.... still it certainly can be helped a bit.  I wouldn't mind it 
spending half an hour of CPU if it needs to.  But I just interrupted it
with ^C with the following result so far:

real    75m44.374s
user    2m5.318s
sys     0m54.059s

(I should have used /usr/bin/time to see the number of page faults).

> I suspect you'll find that with 1GB or RAM you'll have other 
> performance problems that are more pressing ("git clone" comes to mind 
> ;)

Well... same issue actually.  git-pack-objects spent about 40 secs 
firmly at 100% CPU usage counting objects.

Then it got stuck on:

	remote: Done counting 4111366 objects.

again spending 3% CPU and the rest waiting for IO with the disk 
definitely trashing.  It didn't allocate more than 47% of memory during 
that phase which lasted a few minutes.

Then, the "Indexing 4111366 objects." message appeared and CPU usage 
went up to 6% CPU with 67% memory for pack-objects and 30% CPU and 7% 
memory for index-pack while the rest was spent waiting for IO.  This 
also took maybe two minutes.

And now it reached the "Resolving 3305158 deltas." phase with only 
index-pack on the radar with approx 10% CPU and 19% memory, and the rest 
of the time waiting for IO again.

It has been probably half an our now and the thing is at:

	  21% (710502/3305158) done

So it will work and eventually complete.  And the good news is that the 
worst part performance wise is on the client side.  But it looks like 
we're definitely trashing the kernel buffer cache.


Nicolas