[PATCH v2 0/5] repack: don't lose objects to a ".keep" that appears mid-run

["qeesung via GitGitGadget" <gitgitgadget@gmail.com>]

A concurrent push or fetch can make "git repack -d" delete a pack whose
objects were never copied anywhere, and exit 0. We hit this in production: a
ref pointing at a commit that no longer exists, on git 2.43, and it
reproduces on master.

What happens:

 * repack scans for ".keep" files and decides which packs to delete, then
   spawns pack-objects with --honor-pack-keep, which scans again;
 * in between, an index-pack --keep finishes -- a push migrating its
   quarantine, or a fetch -- and installs a ".keep" next to a pack that
   repack has already decided to delete;
 * pack-objects sees that ".keep" and leaves the pack's objects out; repack
   deletes the pack by its earlier list, with force_delete.

The fix is to stop the two processes from scanning separately: hand
pack-objects the snapshot repack took at startup (5/5). Patches 1-4 are what
5/5 needs to be safe:

 * 1/5: under --stdin-packs=follow, a --keep-pack pack stops the traversal
   like a "^" pack; on-disk ".keep" packs never did.
 * 2/5: the cruft walk goes by a stale kept-pack cache, which
   --honor-pack-keep happened to mask. Pre-existing, reproducible today.
 * 3/5: look --keep-pack names up in a sorted list; it gets long.
 * 4/5: --keep-pack-from-file, since a repository can have more kept packs
   than fit on a command line (32K characters on Windows).

Every fix comes with a test that fails without it; the race itself is
reproduced in t7703 by having a ".keep" appear as pack-objects starts. The
full suite passes, and the series merges cleanly into next and seen.

Changes since v1:

 * Dropped 1/6 (odb: don't remove a ".keep" we never installed). Justin
   Tobler is going to fix the receive-pack side properly, by having the ODB
   transaction create the ".keep" itself at commit time rather than reading
   back what index-pack wrote:
   https://lore.kernel.org/git/aql8Wt2q9RnQpjEC@jtobler--20250820-SHC54/
 * The remaining patches are unchanged apart from renumbering.

Qin ShiCheng (5):
  pack-objects: keep --keep-pack open when following
  pack-objects: reset kept-pack cache for cruft walk
  pack-objects: sort --keep-pack list for lookup
  pack-objects: add --keep-pack-from-file
  repack: tell pack-objects which packs are kept

 Documentation/git-pack-objects.adoc |  8 +++
 builtin/pack-objects.c              | 71 ++++++++++++++++++-----
 builtin/repack.c                    | 15 +++++
 odb/source-packed.h                 |  3 +-
 packfile.c                          |  9 ++-
 packfile.h                          |  7 +++
 repack-filtered.c                   |  3 -
 repack.c                            | 34 ++++++++++-
 repack.h                            | 17 +++++-
 t/t5329-pack-objects-cruft.sh       | 40 +++++++++++++
 t/t5331-pack-objects-stdin.sh       | 87 +++++++++++++++++++++++++++++
 t/t7700-repack.sh                   | 43 ++++++++++++++
 t/t7703-repack-geometric.sh         | 72 ++++++++++++++++++++++++
 13 files changed, 386 insertions(+), 23 deletions(-)


base-commit: 3cb9185f65410273787f74333cc027d2ea5daada
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-2219%2Fqeesung%2Frepack-kept-packs-snapshot-v2
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-2219/qeesung/repack-kept-packs-snapshot-v2
Pull-Request: https://github.com/gitgitgadget/git/pull/2219

Range-diff vs v1:

 1:  932e8e425a < -:  ---------- odb: don't remove a ".keep" we never installed
 2:  9349ea48b0 = 1:  8cf72312c5 pack-objects: keep --keep-pack open when following
 3:  a1b85c0a25 = 2:  77aec8941f pack-objects: reset kept-pack cache for cruft walk
 4:  38070935dc = 3:  b76e06a467 pack-objects: sort --keep-pack list for lookup
 5:  f8e27b7aac = 4:  20a051cfb6 pack-objects: add --keep-pack-from-file
 6:  a18e354e73 = 5:  4684fd8552 repack: tell pack-objects which packs are kept

-- 
gitgitgadget