Re: [JGIT RFC PATCH 2/2] Rewrite WindowCache to be easier to follow and maintain

["Shawn O. Pearce" <spearce@spearce.org>]

"Shawn O. Pearce" <spearce@spearce.org> wrote:
> 
> This rewrite generalizes most of the cache logic into a new class,
...
>  I'm tossing this out there for tonight.  Please don't apply until
>  I give a final yay or nay.
...
>  I'm going to burn this in tonight for about 12 hours by pounding
>  a whole bunch of clients against it.

Yay.

This patch looks solid to me.  

I ran it overnight for 12 hours on a test rig.  4-way Intel(R)
Xeon(R) CPU 5148 @ 2.33GHz with 32 GiB physical memory.  The git://
server process was launched with:

  java -Xmx8192m -classpath jgit \
    org.spearce.jgit.pgm.Main daemon
    --port 8853 --export-all base

Note that is the default WindowCacheConfig.

I ran 8 concurrent clone clients on two 2-way systems, 4 clients
per host.  The clone clients randomly selected between three
repositories on each clone attempt:

  linux-2.6 fork (322M on disk)
  repo.git       (1.6M on disk)
  gerrit.git     (9.3M on disk)

I used C git `git clone --bare ...` for the clients to try and
speed up the client side of the test.

In 12 hours the clients successfully completed a total of 2,456
clones, and appear to have been averaging 10,999 KiB/s at peak on
one test host and 4,969 KiB/s on the other.  So JGit was pushing
somewhere around 63,872 KiB/s.  I'm not sure what the network can
really do here; the test clients are in my office and the server
is in a data center somewhere in the same state.

At peak (when all clients picked the linux-2.6 fork roughly around
the same time) I saw the server JVM approaching 380% CPU utilization.
Not too bad giving that the WindowCacheConfig's default settings are
woefully inadequate for 8 concurrent PackWriters on linux-2.6.

Unfortunately, this quad is largest SMP box I have available.
I'm sure we're wasting CPU spinning through cache entries under load.
I still need to do throughput testing, and that may lead to some
tuning changes.  But the code is cleaner and didn't fall over,
so I say we move ahead and apply this rewrite.

-- 
Shawn.