Re: Gitweb caching: Google Summer of Code project

[Lea Wiemann <lewiemann@gmail.com>]

Jakub Narebski wrote:
> Lately he posted a patch
> implementing projects list caching, in a bit different way from how it
> is done on kernel.org, namely by caching data and not final output:

Thanks for this and all the other pointers.

Caching data and not final output is actually what I'm about to try 
next.  If I'm not mistaken, the HTML output is significantly larger than 
the source (repository) data; however, kernel.org still seems to benefit 
from caching the HTML, rather than letting Linux' page cache cache the 
source data.  That leads me to think that the page cache somehow fails 
to cache the source data properly -- I'm not sure why (wild speculation: 
perhaps because of the pack format).  Anyway, I'd hope that I can 
encapsulate the 30-40 git_cmd calls in gitweb.perl and somehow cache 
their results (or, to save memory, the parts of their results that are 
actually used) and cache them using memcached.  If that works well, we 
can stop bothering about frontend (HTML) caching, unless CPU becomes an 
issue, since all HTML pages are generated from cacheable source data.

I'm *kindof* hoping that in the end there will be only few issues with 
cache expiry, since most calls are uniquely identified through hashes. 
(And the ones that are not, like getting the hash of the most recent 
commit, can perhaps be cached with some fairly low expiry time.)

So that's what I'll try next.  If you have any comments or warnings off 
the top of your heads, feel free to send email of course. :)

> the main culprit of [the fork] was splitting gitweb into many, many
> files.  While it helped John in understanding gitweb, it made it
> difficult to merge changes back to mainline.

Interesting point, thanks for letting me know.  (I might have gone ahead 
and tried to split the mainline gitweb myself... ^^)  I think it would 
be nice if gitweb.perl could be split at some point, but I assume there 
are too many patches out there for that to be worth the merge problems, 
right?

-- Lea