[RFC PATCHv4 00/17] gitweb: Simple file based output caching

[Jakub Narebski <jnareb@gmail.com>]

This 17 patches long patch series is intended (yet again) as preliminary
version for splitting large 'gitweb: File based caching layer (from
git.kernel.org)' mega-patch by John 'Warthog9' Hawley aka J.H., by starting
small and adding features piece by piece.

This is fourth version (fourth release) of this series, and is available on
http://repo.or.cz/w/git/jnareb-git.git as 'gitweb/cache-kernel-v4' branch.
Earlier versions are available there as branches 'gitweb/cache-kernel',
'gitweb/cache-kernel-v2' and 'gitweb/cache-kernel-v3'.  Previous version of
this series was sent to git mailing list as:

* [RFC PATCHv3 00/10] gitweb: Simple file based output caching
  Message-Id: <1266349005-15393-1-git-send-email-jnareb@gmail.com>
  http://thread.gmane.org/gmane.comp.version-control.git/140138

This version is based on top of c0cb4ed (git-instaweb: Configure it to work
with new gitweb structure, 2010-05-28), which was merged as 'ps/gitweb-soc'
into next in 5db4adf, on 2010-06-03.  Next version would be rebased on top
of 'next', which would include "sub run" code in 'jn/gitweb-fastcgi', which
would require resolving conflict with this series.

This series of patches does not include comments on individual patches,
comparing them to previous version, and to original series by J.H.  The
next release of this series, hopefully not as RFC, would include them.


The MAIN DIFFERENCE from previous version is that, inspired by splitting
gitweb part of "Splitting gitweb and developing write functionalities
(Integrated web client for git)" Google Summer of Code 2010 project by
Pavan Kumar Sankara, the gitweb output caching code now resides in separate
modules under 'lib/' subdirectory, instead of being all in 'cache.pm' file.

The 'gitweb: href(..., -path_info => 0|1)' which began v3 series is already
merged in, together with other patches from 'jn/gitweb-caching-prep'.

The following changes are available in the git repository at:

  git://repo.or.cz/git/jnareb-git.git gitweb/cache-kernel-v4


All comments welcome!

Shortlog:
~~~~~~~~~
 1. gitweb: Return or exit after done serving request

    This patch was generated in response to noticing the following kind of
    errors in /var/log/httpd/error_log:

      [Sun Jun 13 11:58:02 2010] gitweb.cgi: Subroutine git_atom redefined
      at /var/www/perl/gitweb/gitweb.cgi line 6804.

    when running gitweb under mod_perl (with ModPerl::Registry handler).
    After this patch I didn't see such errors... but I failed to reproduce
    this error without this patch.

    This patch was sent separately to git mailing list as
      http://thread.gmane.org/gmane.comp.version-control.git/149040
    but should probably be skipped.

 2. gitweb: Fix typo in hash key name in %opts in git_header_html

    This patch was sent separately to git mailing list and is already
    present in 'master' as ad709ea (gitweb: Fix typo in hash key name in
    %opts in git_header_html, 2010-06-13).

 3. gitweb: Prepare for splitting gitweb

    IMPORTANT!!!

    This patch is the main reason this series is CC-ed to Pavan Kumar
    Sankara (student) and Christian Couder (mentor) of "Integrated web
    client for git" GSoC2010 project, which includes splitting gitweb.

    As you can see later in series it makes it very easy to add new gitweb
    modules which are to be installed.  Just add

      GITWEB_MODULES += lib/Module/Foo.pm

    to gitweb/Makefile for Module::Foo.

 4. gitweb/lib - Very simple file based cache
 5. gitweb/lib - Stat-based cache expiration

    The major change is that the code is now in GitwebCache::SimpleFileCache
    module in gitweb/lib/GitwebCache/SimpleFileCache.pm file, instead of
    being in gitweb/cache.pm file.

    Other change is removing using File::Spec to manipulate pathnames,
    using faster concatenation instead.  Using File::Spec do not bring us
    extra portability, as gitweb itself assumes '/' as path separator.
    It should be tiny little bit faster without File::Spec::catfile etc.

 6. gitweb/lib - Benchmarking GitwebCache::SimpleFileCache (in t/9603/)

    This is new patch, not present in previous series.  Note that you have
    to run benchmark manually; it is not run by t9503 test.

    The t/t9503/benchmark_caching_interface.pl includes example benchmark
    results at the end of the file.

 7. gitweb/lib - Simple select(FH) based output capture

    The major change is that the code is now in GitwebCache::Capture and
    GitwebCache::Capture::SelectFH modules which are in *separate* files
    (one file per module), instead of being all in gitweb/cache.pm

 8. gitweb/lib - Alternate ways of capturing output

    This is new patch, not present in previous series.  It includes new
    test (not run by t9503) and new benchmark.

 9. gitweb/lib - Cache captured output (using get/set)

    The major change is that the code is now in GitwebCache::CacheOutput
    module in gitweb/lib/GitwebCache/CacheOutput.pm file.

10. gitweb: Add optional output caching

    "use lib __DIR__.'/lib';" is now added by earlier patch in series.
    It uses new infrastructure instead of 'bag of modules' cache.pm file.

11. gitweb/lib - Adaptive cache expiration time
12. gitweb/lib - Use CHI compatible (compute method) caching interface

    Similar to what was in previous version of this series.

13. gitweb/lib - Use locking to avoid 'cache miss stampede' problem
14. gitweb/lib - Serve stale data when waiting for filling cache
15. gitweb/lib - Regenerate (refresh) cache in background


    The major change is that the code is now in new (not present in
    previous version) GitwebCache::FileCacheWithLocking module (in its own
    file), which is derived from GitwebCache::SimpleFileCache.  This allow
    to choose whether to use locking (and other features that require
    locking) or not.

16. gitweb: Show appropriate "Generating..." page when regenerating cache
17. gitweb: Add startup delay to activity indicator for cache

    No major changes, I think (besides using new infrastructure).

 gitweb/Makefile                                |   10 +
 gitweb/README                                  |   62 ++++
 gitweb/gitweb.perl                             |  333 +++++++++++++++++--
 gitweb/lib/GitwebCache/CacheOutput.pm          |   93 ++++++
 gitweb/lib/GitwebCache/Capture.pm              |   66 ++++
 gitweb/lib/GitwebCache/Capture/PerlIO.pm       |   79 +++++
 gitweb/lib/GitwebCache/Capture/SelectFH.pm     |   82 +++++
 gitweb/lib/GitwebCache/Capture/TiedCapture.pm  |  149 +++++++++
 gitweb/lib/GitwebCache/FileCacheWithLocking.pm |  264 +++++++++++++++
 gitweb/lib/GitwebCache/SimpleFileCache.pm      |  417 ++++++++++++++++++++++++
 t/t9500-gitweb-standalone-no-errors.sh         |   19 +
 t/t9503-gitweb-caching.sh                      |   38 +++
 t/t9503/benchmark_caching_interface.pl         |  132 ++++++++
 t/t9503/benchmark_capture_implementations.pl   |  198 +++++++++++
 t/t9503/test_cache_interface.pl                |  409 +++++++++++++++++++++++
 t/t9503/test_cache_output.pl                   |   66 ++++
 t/t9503/test_capture_interface.pl              |   76 +++++
 17 files changed, 2468 insertions(+), 25 deletions(-)

 create mode 100644 gitweb/lib/GitwebCache/CacheOutput.pm
 create mode 100644 gitweb/lib/GitwebCache/Capture.pm
 create mode 100644 gitweb/lib/GitwebCache/Capture/PerlIO.pm
 create mode 100644 gitweb/lib/GitwebCache/Capture/SelectFH.pm
 create mode 100644 gitweb/lib/GitwebCache/Capture/TiedCapture.pm
 create mode 100644 gitweb/lib/GitwebCache/FileCacheWithLocking.pm
 create mode 100644 gitweb/lib/GitwebCache/SimpleFileCache.pm
 create mode 100755 t/t9503-gitweb-caching.sh
 create mode 100755 t/t9503/benchmark_caching_interface.pl
 create mode 100755 t/t9503/benchmark_capture_implementations.pl
 create mode 100755 t/t9503/test_cache_interface.pl
 create mode 100755 t/t9503/test_cache_output.pl
 create mode 100755 t/t9503/test_capture_interface.pl