Git development
 help / color / mirror / Atom feed
From: Siddharth Shrimali <r.siddharth.shrimali@gmail.com>
To: git@vger.kernel.org
Cc: gitster@pobox.com, christian.couder@gmail.com,
	siddharthasthana31@gmail.com, ttaylorr@openai.com, ps@pks.im,
	johannes.schindelin@gmx.de, l.s.r@web.de,
	r.siddharth.shrimali@gmail.com
Subject: [GSoC PATCH v5 0/6] repack: add --drop-filtered to reclaim space in partial clones
Date: Fri, 14 Aug 2026 01:38:24 +0530	[thread overview]
Message-ID: <20260813200830.84348-1-r.siddharth.shrimali@gmail.com> (raw)
In-Reply-To: <20260810174047.6524-1-r.siddharth.shrimali@gmail.com>

This is v5. v4 was at [1].

Partial clones let you work with large repositories without downloading
every blob up front. Missing blobs are lazily fetched from the promisor
remote on demand. Over time these accumulate locally and there is
currently no safe, built-in way to reclaim that space short of
re-cloning. This series adds that reverse direction: enumerate promisor
blobs over a size threshold, drop them locally, and rely on the existing
lazy-fetch machinery to bring them back transparently when needed.

How it works:
  * Enumerate promisor objects directly (ODB_FOR_EACH_OBJECT_PROMISOR_ONLY)
    and select the blobs exceeding the filter threshold. Every enumerated
    object is a promisor object, so it is recoverable from the promisor
    remote as long as the remote still has it, the same assumption the
    rest of partial clone relies on.
  * Rebuild the promisor pack without the selected blobs, reusing the
    existing repack machinery, so the drop is crash-safe (write, fsync,
    install, then delete the old pack).
  * --dry-run lists the candidates and changes nothing.

The guards refuse to run while a merge, rebase, am, cherry-pick, revert,
or bisect is in progress, and refuse to drop a blob referenced by the
current index. Dropped objects stay recoverable via lazy fetch, so these
are a convenience (avoid pointless re-fetch and a surprising
mid-operation fetch) rather than a correctness measure. Both are skipped
for bare repositories.

Changes since v4:
  * list-objects-filter: fixed the object-size local from "unsigned
    long" to "size_t" to match object_info.sizep - caught by Junio
  * list-objects-filter: made the filter options parameter const and
    dropped the now-unnecessary cast at the call site - caught by Christian
  * repack-filtered: comment fixes
  * squashed the documentation into the patches it describes: the option
    description now lands with "actually drop filtered promisor blobs"
    and the guards paragraph with "add guards for --drop-filtered",
    dropping the separate documentation patch, so the series is now 6
    patches - Suggested by Christian
  * improved the "operation in progress" error message to name the
    operations (merge, rebase, am, cherry-pick, revert, or bisect)
  * swept test comments to be proper sentences and fixed tab or space
    indentation

To do:
  * remote verification: verifying against the remote awaits the
    "remote-object-info" cat-file protocol command
  * recency: a "don't cull recently-fetched objects" rule as another
    selection criterion alongside size
  * drop log: introduce with the error-path change that reads it

[1] https://lore.kernel.org/git/20260810174047.6524-1-r.siddharth.shrimali@gmail.com/

Siddharth Shrimali (6):
  builtin/repack: add --drop-filtered and --dry-run options
  list-objects-filter: add list_objects_filter__filter_oidset()
  repack-promisor: allow excluding objects from the rebuilt promisor
    pack
  builtin/repack: enumerate promisor blobs for --drop-filtered
  builtin/repack: actually drop filtered promisor blobs
  builtin/repack: add guards for --drop-filtered

 Documentation/git-repack.adoc   |  37 +++++++
 builtin/repack.c                | 154 +++++++++++++++++++++++++-
 list-objects-filter.c           |  45 ++++++++
 list-objects-filter.h           |  16 +++
 repack-filtered.c               |  82 ++++++++++++++
 repack-promisor.c               |  15 ++-
 repack.h                        |   8 +-
 t/meson.build                   |   1 +
 t/t7706-repack-drop-filtered.sh | 185 ++++++++++++++++++++++++++++++++
 9 files changed, 536 insertions(+), 7 deletions(-)
 create mode 100755 t/t7706-repack-drop-filtered.sh

-- 
2.54.0


  parent reply	other threads:[~2026-08-13 20:08 UTC|newest]

Thread overview: 71+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-07-16 13:28 [RFC PATCH 0/7] repack: add --drop-filtered to reclaim space in partial clones Siddharth Shrimali
2026-07-16 13:28 ` [RFC PATCH 1/7] builtin/repack.c: add --drop-filtered and --dry-run options Siddharth Shrimali
2026-07-16 21:08   ` Junio C Hamano
2026-07-17 18:00     ` Siddharth Shrimali
2026-07-23 19:31     ` Siddharth Asthana
2026-07-18 12:30   ` Christian Couder
2026-07-20 10:19     ` Siddharth Shrimali
2026-07-16 13:28 ` [RFC PATCH 2/7] list-objects-filter: add list_objects_filter__filter_oidset() Siddharth Shrimali
2026-07-16 13:28 ` [RFC PATCH 3/7] repack-promisor: allow excluding objects from the rebuilt promisor pack Siddharth Shrimali
2026-07-16 13:28 ` [RFC PATCH 4/7] builtin/repack: enumerate promisor blobs for --drop-filtered Siddharth Shrimali
2026-07-16 13:28 ` [RFC PATCH 5/7] t7706: test --drop-filtered enumeration and validation Siddharth Shrimali
2026-07-16 13:28 ` [RFC PATCH 6/7] builtin/repack: actually drop filtered promisor blobs Siddharth Shrimali
2026-07-23 19:42   ` Siddharth Asthana
2026-07-25 19:30     ` Siddharth Shrimali
2026-07-16 13:28 ` [RFC PATCH 7/7] repack-promisor: record dropped objects in a drop log Siddharth Shrimali
2026-07-23 19:41   ` Siddharth Asthana
2026-07-25 19:35     ` Siddharth Shrimali
2026-07-23 19:26 ` [RFC PATCH 0/7] repack: add --drop-filtered to reclaim space in partial clones Siddharth Asthana
2026-07-25 19:23   ` Siddharth Shrimali
2026-07-30 17:41 ` [GSoC PATCH v2 " Siddharth Shrimali
2026-07-30 17:41   ` [GSoC PATCH v2 1/7] builtin/repack.c: add --drop-filtered and --dry-run options Siddharth Shrimali
2026-08-04 21:13     ` Siddharth Asthana
2026-07-30 17:41   ` [GSoC PATCH v2 2/7] list-objects-filter: add list_objects_filter__filter_oidset() Siddharth Shrimali
2026-07-30 17:41   ` [GSoC PATCH v2 3/7] repack-promisor: allow excluding objects from the rebuilt promisor pack Siddharth Shrimali
2026-07-30 17:41   ` [GSoC PATCH v2 4/7] builtin/repack: enumerate promisor blobs for --drop-filtered Siddharth Shrimali
2026-07-30 17:41   ` [GSoC PATCH v2 5/7] builtin/repack: actually drop filtered promisor blobs Siddharth Shrimali
2026-07-30 17:41   ` [GSoC PATCH v2 6/7] builtin/repack: add safety guards for --drop-filtered Siddharth Shrimali
2026-08-04 21:13     ` Siddharth Asthana
2026-07-30 17:41   ` [GSoC PATCH v2 7/7] Documentation/git-repack: document --drop-filtered and --dry-run Siddharth Shrimali
2026-07-31 15:33   ` [GSoC PATCH v2 0/7] repack: add --drop-filtered to reclaim space in partial clones Junio C Hamano
2026-08-01 18:19     ` Siddharth Shrimali
2026-08-02  2:18       ` Junio C Hamano
2026-08-02 11:28         ` Siddharth Shrimali
2026-08-04 21:12   ` Siddharth Asthana
2026-08-06 11:21   ` [GSoC PATCH v3 " Siddharth Shrimali
2026-08-06 11:21     ` [GSoC PATCH v3 1/7] builtin/repack.c: add --drop-filtered and --dry-run options Siddharth Shrimali
2026-08-06 11:21     ` [GSoC PATCH v3 2/7] list-objects-filter: add list_objects_filter__filter_oidset() Siddharth Shrimali
2026-08-06 11:21     ` [GSoC PATCH v3 3/7] repack-promisor: allow excluding objects from the rebuilt promisor pack Siddharth Shrimali
2026-08-06 11:21     ` [GSoC PATCH v3 4/7] builtin/repack: enumerate promisor blobs for --drop-filtered Siddharth Shrimali
2026-08-06 11:22     ` [GSoC PATCH v3 5/7] builtin/repack: actually drop filtered promisor blobs Siddharth Shrimali
2026-08-06 11:22     ` [GSoC PATCH v3 6/7] builtin/repack: add guards for --drop-filtered Siddharth Shrimali
2026-08-06 11:22     ` [GSoC PATCH v3 7/7] Documentation/git-repack: document --drop-filtered and --dry-run Siddharth Shrimali
2026-08-06 21:34     ` [GSoC PATCH v3 0/7] repack: add --drop-filtered to reclaim space in partial clones Junio C Hamano
2026-08-06 22:19     ` Junio C Hamano
2026-08-07  9:06       ` Siddharth Shrimali
2026-08-07 20:52         ` Junio C Hamano
2026-08-08 20:07           ` Siddharth Shrimali
2026-08-10 17:40     ` [GSoC PATCH v4 " Siddharth Shrimali
2026-08-10 17:40       ` [GSoC PATCH v4 1/7] builtin/repack.c: add --drop-filtered and --dry-run options Siddharth Shrimali
2026-08-10 17:40       ` [GSoC PATCH v4 2/7] list-objects-filter: add list_objects_filter__filter_oidset() Siddharth Shrimali
2026-08-12 17:09         ` Christian Couder
2026-08-12 18:04         ` Junio C Hamano
2026-08-12 19:22           ` Siddharth Shrimali
2026-08-10 17:40       ` [GSoC PATCH v4 3/7] repack-promisor: allow excluding objects from the rebuilt promisor pack Siddharth Shrimali
2026-08-10 17:40       ` [GSoC PATCH v4 4/7] builtin/repack: enumerate promisor blobs for --drop-filtered Siddharth Shrimali
2026-08-12 17:21         ` Christian Couder
2026-08-10 17:40       ` [GSoC PATCH v4 5/7] builtin/repack: actually drop filtered promisor blobs Siddharth Shrimali
2026-08-10 17:40       ` [GSoC PATCH v4 6/7] builtin/repack: add guards for --drop-filtered Siddharth Shrimali
2026-08-12 17:41         ` Christian Couder
2026-08-12 19:45           ` Siddharth Shrimali
2026-08-10 17:40       ` [GSoC PATCH v4 7/7] Documentation/git-repack: document --drop-filtered and --dry-run Siddharth Shrimali
2026-08-12 17:51         ` Christian Couder
2026-08-11 17:50       ` [GSoC PATCH v4 0/7] repack: add --drop-filtered to reclaim space in partial clones Junio C Hamano
2026-08-12  7:35         ` Siddharth Shrimali
2026-08-13 20:08       ` Siddharth Shrimali [this message]
2026-08-13 20:08         ` [GSoC PATCH v5 1/6] builtin/repack: add --drop-filtered and --dry-run options Siddharth Shrimali
2026-08-13 20:08         ` [GSoC PATCH v5 2/6] list-objects-filter: add list_objects_filter__filter_oidset() Siddharth Shrimali
2026-08-13 20:08         ` [GSoC PATCH v5 3/6] repack-promisor: allow excluding objects from the rebuilt promisor pack Siddharth Shrimali
2026-08-13 20:08         ` [GSoC PATCH v5 4/6] builtin/repack: enumerate promisor blobs for --drop-filtered Siddharth Shrimali
2026-08-13 20:08         ` [GSoC PATCH v5 5/6] builtin/repack: actually drop filtered promisor blobs Siddharth Shrimali
2026-08-13 20:08         ` [GSoC PATCH v5 6/6] builtin/repack: add guards for --drop-filtered Siddharth Shrimali

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20260813200830.84348-1-r.siddharth.shrimali@gmail.com \
    --to=r.siddharth.shrimali@gmail.com \
    --cc=christian.couder@gmail.com \
    --cc=git@vger.kernel.org \
    --cc=gitster@pobox.com \
    --cc=johannes.schindelin@gmx.de \
    --cc=l.s.r@web.de \
    --cc=ps@pks.im \
    --cc=siddharthasthana31@gmail.com \
    --cc=ttaylorr@openai.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox