[PATCH 0/7] pack-revindex: enable on-disk reverse indexes by default

[Taylor Blau <me@ttaylorr.com>]

This series enables pack reverse-indexes to be written to disk by
default instead of computed on-the-fly in memory.

For repositories with large packs, this can have a significant benefit
for both end-users and Git forges. Extensive performance tests appear in
the fifth and sixth commit, but here are some highlights:

  - The time it takes to generate a pack containing objects for the 10
    most-recent commits in linux.git (starting from 68047c48b228) drops
    from ~540ms to ~240ms, yielding a ~2.22x improvement.

  - The time it takes to compute the on-disk object size of a single
    object drops from ~300ms to 4ms, yielding a ~76.96x improvement.

Reverse indexes are a trade-off between the time they take to be
computed versus the time it takes to access a single record. In-memory
reverse indexes take time to generate proportional to the number of
packed objects in the repository, but have instantaneous lookup time.

On-disk reverse indexes can be "generated" instantly (the time it takes
to generate them is a one-time cost, loading them is instantaneous).
But individual record lookup time is slower, since it involves multiple
disk I/O operations.

In the vast majority of cases, this trade-off favors the on-disk ".rev"
files. But in certain cases, the in-memory variant performs more
favorably. Since these cases are narrow, and performance is machine- and
repository-dependent, this series also introduces a new configuration
option to disable reading ".rev" files in the third commit.

The series is structured as follows:

  - A couple of cleanup patches to plug a leak in stage_tmp_packfiles().
  - Three patches to enable `pack.readReverseIndex`.
  - Another patch to change the default of `pack.writeReverseIndex` from
    "false" to "true".
  - A final patch to enable the test suite to be run in a mode that does
    not use on-disk ".rev" files.

Thanks in advance for your review. I'm really excited to get this in the
hands of users after a couple of years of running this at GitHub (and
being opt-in otherwise).

Taylor Blau (7):
  pack-write.c: plug a leak in stage_tmp_packfiles()
  t5325: mark as leak-free
  pack-revindex: make `load_pack_revindex` take a repository
  pack-revindex: introduce GIT_TEST_REV_INDEX_DIE_ON_DISK
  pack-revindex: introduce `pack.readReverseIndex`
  config: enable `pack.writeReverseIndex` by default
  t: invert `GIT_TEST_WRITE_REV_INDEX`

 Documentation/config/pack.txt     |  8 +++++++-
 builtin/index-pack.c              |  5 +++--
 builtin/pack-objects.c            |  5 +++--
 ci/run-build-and-tests.sh         |  1 -
 pack-bitmap.c                     |  5 +++--
 pack-revindex.c                   | 12 +++++++++---
 pack-revindex.h                   |  6 ++++--
 pack-write.c                      |  2 ++
 packfile.c                        |  2 +-
 repo-settings.c                   |  1 +
 repository.h                      |  1 +
 t/README                          |  2 +-
 t/perf/p5312-pack-bitmaps-revs.sh |  3 +--
 t/t5325-reverse-index.sh          | 16 +++++++++++++++-
 14 files changed, 51 insertions(+), 18 deletions(-)

-- 
2.40.0.323.g9c80379958