Re: What's cooking in git.git (Sep 2023, #05; Fri, 15)

[Taylor Blau <me@ttaylorr.com>]

On Fri, Sep 15, 2023 at 06:41:15PM -0700, Junio C Hamano wrote:
> * tb/repack-existing-packs-cleanup (2023-09-13) 8 commits
>   (merged to 'next' on 2023-09-14 at bb8065e89c)
>  + builtin/repack.c: extract common cruft pack loop
>  + builtin/repack.c: avoid directly inspecting "util"
>  + builtin/repack.c: store existing cruft packs separately
>  + builtin/repack.c: extract `has_existing_non_kept_packs()`
>  + builtin/repack.c: extract redundant pack cleanup for existing packs
>  + builtin/repack.c: extract redundant pack cleanup for --geometric
>  + builtin/repack.c: extract marking packs for deletion
>  + builtin/repack.c: extract structure to store existing packs
>
>  The code to keep track of existing packs in the repository while
>  repacking has been refactored.
>
>  Will merge to 'master'.
>  source: <cover.1694632644.git.me@ttaylorr.com>

Nice, I'm happy to see that this is moving along. Thanks, everybody, for
participating in the review :-). It's very nice to have a second set of
eyes when touching the repack machinery.

> * cc/repack-sift-filtered-objects-to-separate-pack (2023-09-11) 9 commits
>  . gc: add `gc.repackFilterTo` config option
>  . repack: implement `--filter-to` for storing filtered out objects
>  . gc: add `gc.repackFilter` config option
>  . repack: add `--filter=<filter-spec>` option
>  . pack-bitmap-write: rebuild using new bitmap when remapping
>  . repack: refactor finding pack prefix
>  . repack: refactor finishing pack-objects command
>  . t/helper: add 'find-pack' test-tool
>  . pack-objects: allow `--filter` without `--stdout`
>
>  "git repack" machinery learns to pay attention to the "--filter="
>  option.
>
>  May need to wait until tb/repack-existing-packs-cleanup stablizes.
>  source: <20230911150618.129737-1-christian.couder@gmail.com>

I sent Christian a draft of what I think the conflict resolution should
look like when rebasing on top of tb/repack-existing-packs-cleanup [1].

I've looked over all but one of the previous rounds, and have seen the
most recent round and am happy with the result. So I think that this
could go in relatively quickly following merging the above.

I have a couple of other repacking-related topics that are in my queue,
but I've been holding off on sending them until Christian's series has
stabilized. They are:

  - tb/cruft-max-size (git@github.com:ttaylorr/git.git)
  - tb/cruft-preferred-inference (git@github.com:ttaylorr/git.git)

The former is on the list at [2], and needs a bit of work before
queueing again. The latter isn't on the list, but fixes a performance
bug where it is possible in rare circumstances to select the cruft pack
as preferred when generating a MIDX bitmap during repacking.

The second isn't crucial to get in any time soon, but I would love to
get it off of my queue before we start cutting pre-releases. At this
rate, I doubt it will be a problem.

[1]: https://lore.kernel.org/git/ZQNKkn0YYLUyN5Ih@nand.local/
[2]: https://lore.kernel.org/git/cover.1694123506.git.me@ttaylorr.com/

> * tb/path-filter-fix (2023-08-30) 15 commits
>  - bloom: introduce `deinit_bloom_filters()`
>  - commit-graph: reuse existing Bloom filters where possible
>  - object.h: fix mis-aligned flag bits table
>  - commit-graph: drop unnecessary `graph_read_bloom_data_context`
>  - commit-graph.c: unconditionally load Bloom filters
>  - t/t4216-log-bloom.sh: harden `test_bloom_filters_not_used()`
>  - bloom: prepare to discard incompatible Bloom filters
>  - bloom: annotate filters with hash version
>  - commit-graph: new filter ver. that fixes murmur3
>  - repo-settings: introduce commitgraph.changedPathsVersion
>  - t4216: test changed path filters with high bit paths
>  - t/helper/test-read-graph: implement `bloom-filters` mode
>  - bloom.h: make `load_bloom_filter_from_graph()` public
>  - t/helper/test-read-graph.c: extract `dump_graph_info()`
>  - gitformat-commit-graph: describe version 2 of BDAT
>
>  The Bloom filter used for path limited history traversal was broken
>  on systems whose "char" is unsigned; update the implementation and
>  bump the format version to 2.
>
>  Needs more work.
>  cf. <20230830200218.GA5147@szeder.dev>
>  source: <cover.1693413637.git.jonathantanmy@google.com>

I think that Jonathan's most recent round of this is ready to get merged
up, cf. [3]. The outstanding issue you note in
<20230830200218.GA5147@szeder.dev> can be addressed separately, I
believe. To that end, I have a RFC-level patch proposed here [4].

[3]: https://lore.kernel.org/git/xmqqo7io8gmo.fsf@gitster.g/
[4]: https://lore.kernel.org/git/ZQnmTXUO94%2FQy8mq@nand.local/

Thanks,
Taylor