Trac+Git: rev-list with pathspec performance?

[Stephen Bash <bash@genarts.com>]

Hi all-

I'm trying to improve the performance of Trac [1], the GitPlugin for Trac[2], and Git.  Trac is being extremely sluggish while browsing source, and profiling revealed the majority of the time was the GitPlugin calling git rev-list.  When I directly entered the rev-list calls from the shell, I found Git itself was performing slower than I would expect...

The bottleneck is while Trac is populating the "last change to file" column in the source browser (see the "rev" column of [3] for an *cough* SVN *cough* example).  This concept of "find the last change to a file" was discussed a few weeks ago [4], but unlike that thread, the GitPlugin is simply calling
   git rev-list --max-count=1 branchName -- fileName
for each file in the current directory.  For files modified recently this is very fast (thousandths of a second), but for older files rev-list takes a long time to come up with an answer (~2-3 seconds on our server).  

I created a script [5] that reproduces the rev-list behavior with 10k commits (ours is about 17k) and 500 files (we peaked at just under 600 in the root of our repo -- that's been cleaned up in the current version, but the history is still there).  On our system the test script fast case is:
   real 0m0.003s, user 0m0.000s, sys 0m0.010s
The slow case is
   real 0m1.072s, user 0m1.050s, sys 0m0.000s

If I naively profile Git I find the worst time offender is tree_entry_interesting with over 10 million calls in the slow case.  That seems high (even every commit, every file would be 500*10000=5 million), but I don't know anything about the actual search algorithm.

Is there anything obvious I can do about this performance bottleneck or is it just the nature of our repository?  Is there potentially a bug in how rev-list works with a pathspec?  Is there a more efficient way to obtain the last commit that changed each file in a directory?  (A hack I'm currently testing is just always return the current commit when Trac asks for the last change... that speeds things up but changes the user experience)

Thanks,
Stephen

References:
[1] http://trac.edgewall.org
[2] http://trac-hacks.org/wiki/GitPlugin
[3] http://trac.edgewall.org/browser/trunk
[4] http://article.gmane.org/gmane.comp.version-control.git/150183/
[5]
#!/bin/bash

git init big-repo
cd big-repo
touch foo
touch bar
for ii in {1..500}
do
   # create some files for background noise
   touch $ii
done
git add .
git commit -qm "initial import"

for ii in {1..10000}
do
   echo "Creating commit $ii"
   echo $ii >> foo
   git add foo
   git commit -qm "simple change $ii"

   if [ $(( $ii % 250 )) == 0 ]
   then
      echo "Running git gc ($ii)"
      git gc --quiet
   fi
done

# Fast case (last change is close to HEAD)
echo "git rev-list --max-count=1 HEAD -- foo ..."
time git rev-list --max-count=1 HEAD -- foo

# Slow case (last change is long before HEAD)
echo "git rev-list --max-count=1 HEAD -- bar ..."
time git rev-list --max-count=1 HEAD -- bar

cd ..

#rm -rf big-repo