[PATCH 00/49] repack: prepare for incremental MIDX-based repacking

Taylor Blau
2025-09-28T22:07:13Z

Note(s) to the maintainer:

* This series depends on ps/packfile-store at dd52a29b78 (packfile:
refactor `get_packed_git_mru()` to work on packfile store,
2025-09-23).

* This series is split out of my local copy of
tb/incremental-midx-part-3. I suggest queueing it as
tb/incremental-midx-part-3.1 ;-).

When developing an incremental MIDX/bitmap-based repacking strategy, I
thought that it might be useful to clean up builtin/repack.c before
adding any amount of new functionality.

At more than 1,700+ lines of code, the 'repack' builtin has grown quite
unwieldy over the years. It has quite a bit of repack-related
functionality, all implemented within the builtin itself. It knows how
to: manage cruft packs, perform geometric repacking, write new MIDXs,
repack promisor objects, keep track of existing packs, and much more.

This series moves the vast majority of the builtin's functionality into
several new compilation units, all grouped under a new repack.h API:

* repack.c
* repack-cruft.c
* repack-filtered.c
* repack-geometry.c
* repack-midx.c
* repack-promisor.c

Moving this code around was less than straightforward, hence the length
of this series. Likewise, since the repacking code can be rather
fragile, I chose to go slower here than I otherwise would have (e.g.,
almost all code movement changes are strict moves with no rewrites in
between).

The structure of the series is roughly as follows:

- The first 10 patches reduce the usage of the_repository, and
the_hash_algo. They don't quite get us to being able to compile
without USE_THE_REPOSITORY_VARIABLE for reasons described in the last
commit of this section.

- The next four commits introduces the new repack.o compilation unit,
and moves a fairly isolated component of the repack builtin.

- The next five commits move the existing_packs API from out of the
builtin into the repack.h API.

- The next five move the generated_pack API.

- The next two move the repack_promisor_objects() API.

- The next four move the pack_geometry API.

- The next nine move various MIDX-related functionality out of the
builtin.

- The next nine move remaining functions that write packs out of the
repack builtin.

- The final commit cleans up the list of `#include`s within the
builtin.

My hope is that this produces a series that is easier to review, despite
its rather long length. Each individual commit tries to do as little as
possible, and most will benefit from being reviewed with --color-moved.
The result, in my opinion, is a 'builtin/repack.c' is much cleaner than
it was when we started.

Nothing in this series is a strict requirement for implementing a
repacking strategy based on incremental MIDX/bitmaps. But since the new
functionality required for that is lengthy in and of itself, having a
clean slate to start from seemed worthwhile to me.

Thanks in advance for your review!

Taylor Blau (49):
builtin/repack.c: avoid "the_repository" in `cmd_repack()`
builtin/repack.c: avoid "the_repository" in existing packs API
builtin/repack.c: avoid "the_repository" when taking a ref snapshot
builtin/repack.c: avoid "the_repository" when removing packs
builtin/repack.c: avoid "the_repository" when repacking promisor
objects
builtin/repack.c: avoid "the_hash_algo" when deleting packs
builtin/repack.c: avoid "the_hash_algo" in `write_oid()`
builtin/repack: avoid "the_hash_algo" in `repack_promisor_objects()`
builtin/repack.c: avoid "the_hash_algo" in `finish_pack_objects_cmd()`
builtin/repack.c: avoid using `hash_to_hex()` in pack geometry
repack: introduce new compilation unit
builtin/repack.c: pass both pack_objects args to repack_config
repack: move 'delta_base_offset' to 'struct pack_objects_args'
repack: remove 'prepare_pack_objects' from the builtin
builtin/repack.c: rename many 'struct existing_packs' functions
repack: remove 'remove_redundant_pack' from the builtin
builtin/repack.c: pass "packdir" when removing packs
builtin/repack.c: avoid unnecessary numeric casts in existing_packs
repack: remove 'existing_packs' API from the builtin
builtin/repack.c: rename "struct generated_pack_data"
builtin/repack.c: factor our "generated_pack_install"
builtin/repack.c: pass "packtmp" to `generated_pack_populate()`
builtin/repack.c: provide pack locations to `generated_pack_install()`
repack: remove 'generated_pack' API from the builtin
builtin/repack.c: pass "packtmp" to `repack_promisor_objects()`
builtin/repack.c: remove "repack_promisor_objects()" from the builtin
builtin/repack.c: rename various pack_geometry functions
builtin/repack.c: pass 'pack_kept_objects' to `pack_geometry_init()`
builtin/repack.c: pass 'packdir' to `pack_geometry_remove_redundant()`
repack: remove pack_geometry API from the builtin
builtin/repack.c: remove ref snapshotting from builtin
builtin/repack.c: extract opts struct for
'write_midx_included_packs()'
builtin/repack.c: use a string_list for 'midx_pack_names'
repack: keep track of MIDX pack names using existing_packs
builtin/repack.c: reorder `remove_redundant_bitmaps()`
builtin/repack.c: inline `remove_redundant_bitmaps()`
builtin/repack.c: pass `repack_write_midx_opts` to
`midx_included_packs`
builtin/repack.c: inline packs within `write_midx_included_packs()`
repack: 'write_midx_included_packs' API from the builtin
builtin/repack.c: introduce `struct write_pack_opts`
builtin/repack.c: use `write_pack_opts` within `write_cruft_pack()`
repack: move `find_pack_prefix()` out of the builtin
repack: extract `write_pack_opts_is_local()`
builtin/repack.c: pass `write_pack_opts` to
`finish_pack_objects_cmd()`
repack: move `finish_pack_objects_cmd()` out of the builtin
repack: move `pack_kept_objects` to `struct pack_objects_args`
repack: move `write_filtered_pack()` out of the builtin
repack: move `write_cruft_pack()` out of the builtin
builtin/repack.c: clean up unused `#include`s

Makefile | 6 +
builtin/repack.c | 1348 +++------------------------------------------
meson.build | 6 +
repack-cruft.c | 99 ++++
repack-filtered.c | 51 ++
repack-geometry.c | 232 ++++++++
repack-midx.c | 372 +++++++++++++
repack-promisor.c | 102 ++++
repack.c | 361 ++++++++++++
repack.h | 146 +++++
10 files changed, 1461 insertions(+), 1262 deletions(-)
create mode 100644 repack-cruft.c
create mode 100644 repack-filtered.c
create mode 100644 repack-geometry.c
create mode 100644 repack-midx.c
create mode 100644 repack-promisor.c
create mode 100644 repack.c
create mode 100644 repack.h

base-commit: dd52a29b78d80e425be660f3b443a42e0374a7d1
--
2.51.0.243.g16eca91f2c0