[PATCH 0/2] Fix background maintenance regression in Git 2.45.0

["Derrick Stolee via GitGitGadget" <gitgitgadget@gmail.com>]

Here is an issue I noticed while exploring issues with my local copy of a
large monorepo. I was intending to show some engineers how nice the objects
were maintained by background maintenance, but saw hundreds of small
pack-files that were up to two months old. This time matched when I upgraded
to the microsoft/git fork that included the 2.45.0 release of Git.

The issue is that 'git multi-pack-index repack' was taught to call 'git
pack-objects' with the new '--stdin-packs' option. However, this changes the
object selection algorithm. Instead of using the objects referenced by the
multi-pack-index, it compares pack-files using a list of "included" and
"excluded" pack-files. This loses some granularity of how the
multi-pack-index chooses among duplicate objects.

The end result is that some objects that would normally have been included
in the new pack-file are no longer included. The copy that the
multi-pack-index references is in the pack-file that was intended to be
repacked, so that pack-file cannot be expired in the next 'git
multi-pack-index expire' step and is included again in the batch of objects
to repack.

In the context of the change that is reverted by this series, it seems the
motivation of the change was two-fold:

 1. some I/O benefits to using pack names over object names, and
 2. the ability to use an object walk to improve delta compression.

In my local prototyping, I've found that we could improve 'git pack-objects'
to use an object walk when given a set of objects over stdin without needing
to use pack-file names. I do not believe the '--stdin-packs' option should
be used for the 'git multi-pack-index repack' mechanism, or at least should
be done with great care and only in specific cases where some assumptions
can be made around duplicate objects and closure under reachability.

However, the prototype I've built to get these benefits is non-trivial due
to working to guarantee that partial clones do not accidentally download
missing blobs. That will follow in a separate series that can be reviewed at
a slower pace.

Thanks, -Stolee

Derrick Stolee (2):
  t5319: add failing test case for repack/expire
  midx-write: revert use of --stdin-packs

 midx-write.c                | 18 ++++++------
 t/t5319-multi-pack-index.sh | 55 +++++++++++++++++++++++++++++++++++++
 2 files changed, 64 insertions(+), 9 deletions(-)


base-commit: bea9ecd24b0c3bf06cab4a851694fe09e7e51408
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1764%2Fderrickstolee%2Fincremental-repack-fix-v1
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1764/derrickstolee/incremental-repack-fix-v1
Pull-Request: https://github.com/gitgitgadget/git/pull/1764
-- 
gitgitgadget