[PATCH 00/49] repack: prepare for incremental MIDX-based repacking

[Taylor Blau <me@ttaylorr.com>]

Note(s) to the maintainer:

 * This series depends on ps/packfile-store at dd52a29b78 (packfile:
   refactor `get_packed_git_mru()` to work on packfile store,
   2025-09-23).

 * This series is split out of my local copy of
   tb/incremental-midx-part-3. I suggest queueing it as
   tb/incremental-midx-part-3.1 ;-).

When developing an incremental MIDX/bitmap-based repacking strategy, I
thought that it might be useful to clean up builtin/repack.c before
adding any amount of new functionality.

At more than 1,700+ lines of code, the 'repack' builtin has grown quite
unwieldy over the years. It has quite a bit of repack-related
functionality, all implemented within the builtin itself. It knows how
to: manage cruft packs, perform geometric repacking, write new MIDXs,
repack promisor objects, keep track of existing packs, and much more.

This series moves the vast majority of the builtin's functionality into
several new compilation units, all grouped under a new repack.h API:

 * repack.c
 * repack-cruft.c
 * repack-filtered.c
 * repack-geometry.c
 * repack-midx.c
 * repack-promisor.c

Moving this code around was less than straightforward, hence the length
of this series. Likewise, since the repacking code can be rather
fragile, I chose to go slower here than I otherwise would have (e.g.,
almost all code movement changes are strict moves with no rewrites in
between).

The structure of the series is roughly as follows:

 - The first 10 patches reduce the usage of the_repository, and
   the_hash_algo. They don't quite get us to being able to compile
   without USE_THE_REPOSITORY_VARIABLE for reasons described in the last
   commit of this section.

 - The next four commits introduces the new repack.o compilation unit,
   and moves a fairly isolated component of the repack builtin.

 - The next five commits move the existing_packs API from out of the
   builtin into the repack.h API.

 - The next five move the generated_pack API.

 - The next two move the repack_promisor_objects() API.

 - The next four move the pack_geometry API.

 - The next nine move various MIDX-related functionality out of the
   builtin.

 - The next nine move remaining functions that write packs out of the
   repack builtin.

 - The final commit cleans up the list of `#include`s within the
   builtin.

My hope is that this produces a series that is easier to review, despite
its rather long length. Each individual commit tries to do as little as
possible, and most will benefit from being reviewed with --color-moved.
The result, in my opinion, is a 'builtin/repack.c' is much cleaner than
it was when we started.

Nothing in this series is a strict requirement for implementing a
repacking strategy based on incremental MIDX/bitmaps. But since the new
functionality required for that is lengthy in and of itself, having a
clean slate to start from seemed worthwhile to me.

Thanks in advance for your review!

Taylor Blau (49):
  builtin/repack.c: avoid "the_repository" in `cmd_repack()`
  builtin/repack.c: avoid "the_repository" in existing packs API
  builtin/repack.c: avoid "the_repository" when taking a ref snapshot
  builtin/repack.c: avoid "the_repository" when removing packs
  builtin/repack.c: avoid "the_repository" when repacking promisor
    objects
  builtin/repack.c: avoid "the_hash_algo" when deleting packs
  builtin/repack.c: avoid "the_hash_algo" in `write_oid()`
  builtin/repack: avoid "the_hash_algo" in `repack_promisor_objects()`
  builtin/repack.c: avoid "the_hash_algo" in `finish_pack_objects_cmd()`
  builtin/repack.c: avoid using `hash_to_hex()` in pack geometry
  repack: introduce new compilation unit
  builtin/repack.c: pass both pack_objects args to repack_config
  repack: move 'delta_base_offset' to 'struct pack_objects_args'
  repack: remove 'prepare_pack_objects' from the builtin
  builtin/repack.c: rename many 'struct existing_packs' functions
  repack: remove 'remove_redundant_pack' from the builtin
  builtin/repack.c: pass "packdir" when removing packs
  builtin/repack.c: avoid unnecessary numeric casts in existing_packs
  repack: remove 'existing_packs' API from the builtin
  builtin/repack.c: rename "struct generated_pack_data"
  builtin/repack.c: factor our "generated_pack_install"
  builtin/repack.c: pass "packtmp" to `generated_pack_populate()`
  builtin/repack.c: provide pack locations to `generated_pack_install()`
  repack: remove 'generated_pack' API from the builtin
  builtin/repack.c: pass "packtmp" to `repack_promisor_objects()`
  builtin/repack.c: remove "repack_promisor_objects()" from the builtin
  builtin/repack.c: rename various pack_geometry functions
  builtin/repack.c: pass 'pack_kept_objects' to `pack_geometry_init()`
  builtin/repack.c: pass 'packdir' to `pack_geometry_remove_redundant()`
  repack: remove pack_geometry API from the builtin
  builtin/repack.c: remove ref snapshotting from builtin
  builtin/repack.c: extract opts struct for
    'write_midx_included_packs()'
  builtin/repack.c: use a string_list for 'midx_pack_names'
  repack: keep track of MIDX pack names using existing_packs
  builtin/repack.c: reorder `remove_redundant_bitmaps()`
  builtin/repack.c: inline `remove_redundant_bitmaps()`
  builtin/repack.c: pass `repack_write_midx_opts` to
    `midx_included_packs`
  builtin/repack.c: inline packs within `write_midx_included_packs()`
  repack: 'write_midx_included_packs' API from the builtin
  builtin/repack.c: introduce `struct write_pack_opts`
  builtin/repack.c: use `write_pack_opts` within `write_cruft_pack()`
  repack: move `find_pack_prefix()` out of the builtin
  repack: extract `write_pack_opts_is_local()`
  builtin/repack.c: pass `write_pack_opts` to
    `finish_pack_objects_cmd()`
  repack: move `finish_pack_objects_cmd()` out of the builtin
  repack: move `pack_kept_objects` to `struct pack_objects_args`
  repack: move `write_filtered_pack()` out of the builtin
  repack: move `write_cruft_pack()` out of the builtin
  builtin/repack.c: clean up unused `#include`s

 Makefile          |    6 +
 builtin/repack.c  | 1348 +++------------------------------------------
 meson.build       |    6 +
 repack-cruft.c    |   99 ++++
 repack-filtered.c |   51 ++
 repack-geometry.c |  232 ++++++++
 repack-midx.c     |  372 +++++++++++++
 repack-promisor.c |  102 ++++
 repack.c          |  361 ++++++++++++
 repack.h          |  146 +++++
 10 files changed, 1461 insertions(+), 1262 deletions(-)
 create mode 100644 repack-cruft.c
 create mode 100644 repack-filtered.c
 create mode 100644 repack-geometry.c
 create mode 100644 repack-midx.c
 create mode 100644 repack-promisor.c
 create mode 100644 repack.c
 create mode 100644 repack.h


base-commit: dd52a29b78d80e425be660f3b443a42e0374a7d1
-- 
2.51.0.243.g16eca91f2c0