Git development
 help / color / mirror / Atom feed
* [RFC PATCH 0/7] repack: add --drop-filtered to reclaim space in partial clones
@ 2026-07-16 13:28 Siddharth Shrimali
  2026-07-16 13:28 ` [RFC PATCH 1/7] builtin/repack.c: add --drop-filtered and --dry-run options Siddharth Shrimali
                   ` (6 more replies)
  0 siblings, 7 replies; 9+ messages in thread
From: Siddharth Shrimali @ 2026-07-16 13:28 UTC (permalink / raw)
  To: git
  Cc: gitster, christian.couder, siddharthasthana31, me, ps,
	johannes.schindelin, l.s.r, r.siddharth.shrimali

This is an RFC series seeking feedback on the design and approach.
Several pieces are still missing (noted below) and the commit
organization needs cleanup.

Partial clones let you work with large repositories without downloading
every blob up front and the missing blobs are lazily fetched from the promisor
remote on demand. Over time, though, these lazily-fetched blobs
accumulate locally and there is currently no safe, built-in way to
reclaim that disk space instead of re-cloning.

This series adds a "git repack --drop-filtered --filter=<spec>" command
that removes large, locally-held promisor blobs that are recoverable
from the promisor remote. The dropped blobs become absent locally but
remain lazily re-fetchable, making the partial-clone still reversible.

How it works:
  * Enumerate promisor objects directly (ODB_FOR_EACH_OBJECT_PROMISOR_ONLY)
    and select the blobs exceeding the filter threshold. Because every
    enumerated object is a promisor object, it is guaranteed recoverable and
    locally-created objects are never candidates.

  * Rebuild the promisor pack without the selected blobs, reusing the
    existing repack machinery, so the drop is crash-safe.

  * Record each dropped object in a drop log
    ($GIT_DIR/objects/info/promisor-dropped) so a later change can
    explain a failed lazy fetch (when it was dropped, which filter
    matched, which remotes) instead of a bare "could not fetch" error.

  * --dry-run lists the candidates and changes nothing.

Planned follow-ups:
  * Safety guards: refuse to run while a merge/rebase/cherry-pick is in
    progress, and refuse to drop blobs referenced by the current index.

  * Authoritative remote verification: the drop log currently lists all
    configured promisor remotes rather than the exact remote each object
    is recoverable from, because there is no client-side way to query a
    remote for object availability yet. A "remote-object-info" command
    is being added to the "git cat-file --batch" protocol for this. Once
    available, the exact remote can be recorded.

Known issues to address in v2:
  * There is churn between "enumerate promisor blobs" and "actually drop
    filtered promisor blobs". The former introduces
    enumerate_promisor_blobs() with an interim signature that the latter
    rewrites. These will be reorganized so the function is introduced
    in its final form.

  * The tests are in a standalone commit. They will instead be
    distributed into the commits that introduce the behavior they test.

Siddharth Shrimali (7):
  builtin/repack.c: add --drop-filtered and --dry-run options
  list-objects-filter: add list_objects_filter__filter_oidset()
  repack-promisor: allow excluding objects from the rebuilt promisor
    pack
  builtin/repack: enumerate promisor blobs for --drop-filtered
  t7706: test --drop-filtered enumeration and validation
  builtin/repack: actually drop filtered promisor blobs
  repack-promisor: record dropped objects in a drop log

 builtin/repack.c                |  76 ++++++++++++++++-
 list-objects-filter.c           |  45 ++++++++++
 list-objects-filter.h           |  16 ++++
 repack-filtered.c               |  81 ++++++++++++++++++
 repack-promisor.c               | 106 ++++++++++++++++++++++-
 repack.h                        |  12 ++-
 t/meson.build                   |   1 +
 t/t7706-repack-drop-filtered.sh | 145 ++++++++++++++++++++++++++++++++
 8 files changed, 478 insertions(+), 4 deletions(-)
 create mode 100755 t/t7706-repack-drop-filtered.sh

-- 
2.54.0


^ permalink raw reply	[flat|nested] 9+ messages in thread

end of thread, other threads:[~2026-07-16 21:08 UTC | newest]

Thread overview: 9+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-07-16 13:28 [RFC PATCH 0/7] repack: add --drop-filtered to reclaim space in partial clones Siddharth Shrimali
2026-07-16 13:28 ` [RFC PATCH 1/7] builtin/repack.c: add --drop-filtered and --dry-run options Siddharth Shrimali
2026-07-16 21:08   ` Junio C Hamano
2026-07-16 13:28 ` [RFC PATCH 2/7] list-objects-filter: add list_objects_filter__filter_oidset() Siddharth Shrimali
2026-07-16 13:28 ` [RFC PATCH 3/7] repack-promisor: allow excluding objects from the rebuilt promisor pack Siddharth Shrimali
2026-07-16 13:28 ` [RFC PATCH 4/7] builtin/repack: enumerate promisor blobs for --drop-filtered Siddharth Shrimali
2026-07-16 13:28 ` [RFC PATCH 5/7] t7706: test --drop-filtered enumeration and validation Siddharth Shrimali
2026-07-16 13:28 ` [RFC PATCH 6/7] builtin/repack: actually drop filtered promisor blobs Siddharth Shrimali
2026-07-16 13:28 ` [RFC PATCH 7/7] repack-promisor: record dropped objects in a drop log Siddharth Shrimali

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox