[RFC PATCH 00/10] repack: combine '--geometric' and '--cruft'

[Taylor Blau <me@ttaylorr.com>]

First, a short note. This series is an RFC because I have not had the
chance to review and test it as thoroughly as I normally would, and
because we are deep in the -rc phase.

I wanted to get this series off my backlog since I have decided to leave
GitHub at the end of the month for a new role. I will still be
contributing to Git in my new role (which I will start in the early part
of July), but wanted to get this off my backlog nonetheless.

This series teaches `git repack` how to combine `--geometric` and
`--cruft`.

Today these two modes are mutually exclusive, since `--cruft` implies
`-a`, and `-a` is fundamentally incompatible with `--geometric`. As a
result, repositories have to choose between keeping reachable objects in
a geometric progression of packs and collecting unreachable objects into
cruft packs.

The goal of this series is to to be able to do both simultaneously. When
both options are given, 'git repack' rolls up the selected non-cruft
packs as usual while collecting unreachable objects separately into a
cruft pack. That means a command like

    $ git repack -d --geometric=2 --cruft --combine-cruft-below-size=1G

will keep reachable non-cruft packs in a geometric progression, while
combining sufficiently-small cruft packs (along with newly-discovered
unreachable objects) into a fresh cruft pack.

The series is structured roughly as follows:

 * The first two patches prepare the cruft pack machinery for the later
   changes by making non-kept pack exclusion unconditional and
   extracting a helper for looking up packs in an `existing_packs` list.

 * The next four patches route geometric pack deletion through the
   common `existing_packs` machinery. They mark packs above the
   geometric split as retained, teach incremental-MIDX retention not to
   keep packs that are being rolled up, switch geometric repacks over to
   the common deletion path, and then remove the old geometry-specific
   deletion helper.

 * The next three patches teach `pack-objects` the new pieces needed by
   this mode. The main addition is `--stdin-packs=follow-reachable`,
   which walks from reference tips and includes only reachable objects
   from the selected packs, while still allowing traversal through
   excluded-open packs and stopping at excluded-closed ones. The
   following patch teaches that mode to use `--refs-snapshot`, so that
   `pack-objects` and the MIDX bitmap writer can agree on the same set
   of tips.

 * The final patch wires everything together in `git repack`, including
   teaching the cruft writer how to interpret the geometric split when
   choosing which packs to include or exclude.

Thanks in advance for your review!

Taylor Blau (10):
  repack: unconditionally exclude non-kept packs
  repack: extract `locate_existing_pack()` helper
  repack: mark geometric progression of packs as retained
  repack: teach MIDX retention about geometric rollups
  repack: delete geometric packs via existing_packs
  repack-geometry: drop unused redundant-pack removal
  pack-objects: extract `stdin_packs_add_all_pack_entries()`
  pack-objects: introduce '--stdin-packs=follow-reachable'
  pack-objects: support '--refs-snapshot' with 'follow-reachable'
  repack: support combining '--geometric' with '--cruft'

 Documentation/git-pack-objects.adoc |  25 +++
 Documentation/git-repack.adoc       |  11 ++
 builtin/pack-objects.c              | 276 ++++++++++++++++++++++++----
 builtin/repack.c                    |  38 ++--
 repack-cruft.c                      |  29 ++-
 repack-geometry.c                   |  44 -----
 repack.c                            | 101 +++++++++-
 repack.h                            |  15 +-
 t/t5331-pack-objects-stdin.sh       | 201 ++++++++++++++++++++
 t/t7704-repack-cruft.sh             | 251 +++++++++++++++++++++++++
 10 files changed, 878 insertions(+), 113 deletions(-)


base-commit: ab776a62a78576513ee121424adb19597fbb7613
-- 
2.55.0.rc2.10.g29e31820dce