[PATCH v4 00/16] repack: incremental MIDX/bitmap-based repacking

[Taylor Blau <me@ttaylorr.com>]

Here is another small reroll of my series to implement the last
remaining component of the incremental MIDX/bitmap-based repacking
strategy [1].

The changes since v3 are fairly small:

 - `repack_prepare_midx_command()` now calls its third parameter
   "subcommand" instead of "verb".

 - `clear_incremental_midx_files()` now avoids a redundant conditional
   around `r->objects` and reuses a local `struct odb_source *` when
   closing any open incremental MIDX handles.

As usual, a range-diff is included below for convenience. Thanks in
advance for reviewing!

[1]: https://lore.kernel.org/git/cover.1777507303.git.me@ttaylorr.com/

Taylor Blau (16):
  midx-write: handle noop writes when converting incremental chains
  midx: use `strset` for retained MIDX files
  midx: build `keep_hashes` array in order
  midx: use `strvec` for `keep_hashes`
  midx: introduce `--no-write-chain-file` for incremental MIDX writes
  midx: support custom `--base` for incremental MIDX writes
  repack: track the ODB source via existing_packs
  midx: expose `midx_layer_contains_pack()`
  repack-midx: factor out `repack_prepare_midx_command()`
  repack-midx: extract `repack_fill_midx_stdin_packs()`
  repack-geometry: prepare for incremental MIDX repacking
  builtin/repack.c: convert `--write-midx` to an `OPT_CALLBACK`
  packfile: ensure `close_pack_revindex()` frees in-memory revindex
  repack: implement incremental MIDX repacking
  repack: introduce `--write-midx=incremental`
  repack: allow `--write-midx=incremental` without `--geometric`

 Documentation/config/repack.adoc        |  18 +
 Documentation/git-multi-pack-index.adoc |  32 +-
 Documentation/git-repack.adoc           |  44 +-
 builtin/multi-pack-index.c              |  48 +-
 builtin/repack.c                        | 102 +++-
 midx-write.c                            | 206 ++++---
 midx.c                                  | 102 ++--
 midx.h                                  |  11 +-
 packfile.c                              |   2 +
 repack-geometry.c                       |  48 +-
 repack-midx.c                           | 710 +++++++++++++++++++++++-
 repack.c                                |  58 +-
 repack.h                                |  26 +-
 t/meson.build                           |   1 +
 t/t5334-incremental-multi-pack-index.sh |  63 +++
 t/t5335-compact-multi-pack-index.sh     | 113 ++++
 t/t7705-repack-incremental-midx.sh      | 525 ++++++++++++++++++
 17 files changed, 1909 insertions(+), 200 deletions(-)
 create mode 100755 t/t7705-repack-incremental-midx.sh

Range-diff against v3:
 1:  d6c27317c25 =  1:  ead11e610c8 midx-write: handle noop writes when converting incremental chains
 2:  629c8d23116 =  2:  ece55bf2957 midx: use `strset` for retained MIDX files
 3:  e303bf6a4ac =  3:  5609d1941e6 midx: build `keep_hashes` array in order
 4:  42d76c70060 =  4:  13b7c808860 midx: use `strvec` for `keep_hashes`
 5:  2c80aa34fac =  5:  cac3fd54bf0 midx: introduce `--no-write-chain-file` for incremental MIDX writes
 6:  2a05f4b86f3 =  6:  1bbb387d6b6 midx: support custom `--base` for incremental MIDX writes
 7:  92aba3d366f =  7:  4a93adb3ad3 repack: track the ODB source via existing_packs
 8:  d3ac65c1f11 =  8:  8d1b8b1d301 midx: expose `midx_layer_contains_pack()`
 9:  1bd2f194c6f !  9:  42111e5f75d repack-midx: factor out `repack_prepare_midx_command()`
    @@ Commit message
         subcommands), so extract the common portions of the command setup into a
         reusable `repack_prepare_midx_command()` helper.
     
    -    The extracted helper sets `git_cmd`, pushes the `multi-pack-index`
    -    subcommand and verb, and handles `--progress`/`--no-progress` and
    -    `--bitmap` flags. The remaining arguments that are specific to the
    -    `write` subcommand (such as `--stdin-packs`) are left to the caller.
    +    The extracted helper sets `git_cmd`, pushes `multi-pack-index` and a
    +    subcommand, and handles `--progress`/`--no-progress` and `--bitmap`
    +    flags. The remaining arguments that are specific to the `write`
    +    subcommand (such as `--stdin-packs`) are left to the caller.
     
         No functional changes are included in this patch.
     
    @@ repack-midx.c: static void remove_redundant_bitmaps(struct string_list *include,
      
     +static void repack_prepare_midx_command(struct child_process *cmd,
     +					struct repack_write_midx_opts *opts,
    -+					const char *verb)
    ++					const char *subcommand)
     +{
     +	cmd->git_cmd = 1;
     +
    -+	strvec_pushl(&cmd->args, "multi-pack-index", verb, NULL);
    ++	strvec_pushl(&cmd->args, "multi-pack-index", subcommand, NULL);
     +
     +	if (opts->show_progress)
     +		strvec_push(&cmd->args, "--progress");
10:  2a87a1e4561 = 10:  ed76e6efd1c repack-midx: extract `repack_fill_midx_stdin_packs()`
11:  3d32b9c88da = 11:  9665f1b3a64 repack-geometry: prepare for incremental MIDX repacking
12:  1f7a5479bb8 = 12:  e0db62b9f10 builtin/repack.c: convert `--write-midx` to an `OPT_CALLBACK`
13:  b155f25d53c = 13:  c8c846b1ac1 packfile: ensure `close_pack_revindex()` frees in-memory revindex
14:  ef012314930 = 14:  78f1a98c1ea repack: implement incremental MIDX repacking
15:  04cfecd5136 ! 15:  95bf8b21fff repack: introduce `--write-midx=incremental`
    @@ midx.c: void clear_midx_file(struct repository *r)
     +void clear_incremental_midx_files(struct repository *r,
     +				  const struct strvec *keep_hashes)
     +{
    ++	struct odb_source *source = r->objects->sources;
     +	struct strbuf chain = STRBUF_INIT;
     +
    -+	get_midx_chain_filename(r->objects->sources, &chain);
    ++	get_midx_chain_filename(source, &chain);
     +
    -+	if (r->objects) {
    -+		struct odb_source *source = r->objects->sources;
    -+		for (source = r->objects->sources; source; source = source->next) {
    -+			struct odb_source_files *files = odb_source_files_downcast(source);
    -+			if (files->packed->midx)
    -+				close_midx(files->packed->midx);
    -+			files->packed->midx = NULL;
    -+		}
    ++	for (; source; source = source->next) {
    ++		struct odb_source_files *files = odb_source_files_downcast(source);
    ++		if (files->packed->midx)
    ++			close_midx(files->packed->midx);
    ++		files->packed->midx = NULL;
     +	}
     +
     +	if (!keep_hashes && remove_path(chain.buf))
16:  1c05dfce579 = 16:  8bd0ec98dc3 repack: allow `--write-midx=incremental` without `--geometric`

base-commit: 7bcaabddcf68bd0702697da5904c3b68c52f94cf
-- 
2.54.0.175.g8bd0ec98dc3