'git repack' and repack.writeBitmaps=true with kept packs

'git repack' and repack.writeBitmaps=true with kept packs

[Steven Noonan]

I have some unexpected behavior with 'git repack' on git 2.10.2 and 2.11.0.

$ cat /etc/gitconfig
[pack]
        writeBitmapHashCache = true
[repack]
        writeBitmaps = true
$ touch objects/pack/pack-3841d81123a96cedeb3c1bd7acf7e29bfba26639.keep
$ find objects
objects
objects/pack
objects/pack/pack-3841d81123a96cedeb3c1bd7acf7e29bfba26639.keep
objects/pack/pack-3841d81123a96cedeb3c1bd7acf7e29bfba26639.idx
objects/pack/pack-3841d81123a96cedeb3c1bd7acf7e29bfba26639.bitmap
objects/pack/pack-3841d81123a96cedeb3c1bd7acf7e29bfba26639.pack
objects/info
objects/info/packs
$ git repack -Adfl
Counting objects: 16321, done.
Delta compression using up to 24 threads.
Compressing objects: 100% (16118/16118), done.
Writing objects: 100% (16321/16321), done.
Reusing bitmaps: 110, done.
Selecting bitmap commits: 3257, done.
Building bitmaps: 100% (137/137), done.
Total 16321 (delta 11568), reused 4507 (delta 0)
$ du -sh objects/pack/pack*
100K    objects/pack/pack-3841d81123a96cedeb3c1bd7acf7e29bfba26639.bitmap
524K    objects/pack/pack-3841d81123a96cedeb3c1bd7acf7e29bfba26639.idx
512     objects/pack/pack-3841d81123a96cedeb3c1bd7acf7e29bfba26639.keep
4.3M    objects/pack/pack-3841d81123a96cedeb3c1bd7acf7e29bfba26639.pack
100K    objects/pack/pack-6043151bee7bd61bdae6e3a2ba0f13cd1b0277af.bitmap
524K    objects/pack/pack-6043151bee7bd61bdae6e3a2ba0f13cd1b0277af.idx
5.8M    objects/pack/pack-6043151bee7bd61bdae6e3a2ba0f13cd1b0277af.pack

It seems like it's behaving as though I've provided
--pack-kept-objects. In order to ensure the .bitmap is created, it
repacks everything, including everything in existing .pack files (not
respecting .keep). But then it's not deleting the old .pack file
(oddly, respecting .keep).

What I'd expect it to do here is ignore the 'repack.writeBitmaps =
true' value if there's a .keep that needs to be respected. Is this not
a correct assumption?

Re: 'git repack' and repack.writeBitmaps=true with kept packs

[Jeff King]

On Wed, Nov 30, 2016 at 04:15:33PM -0800, Steven Noonan wrote:

> It seems like it's behaving as though I've provided
> --pack-kept-objects. In order to ensure the .bitmap is created, it
> repacks everything, including everything in existing .pack files (not
> respecting .keep). But then it's not deleting the old .pack file
> (oddly, respecting .keep).

Right, that's exactly what's happening.

The bitmaps require a completely reachable set inside the pack, so if
you omit some objects that are in .keep packs, we cannot generate the
bitmap. So we have to either disable bitmaps, or pack the kept objects.
By default, we do the latter (and I'll explain why in a minute).

We can't delete the .keep packfiles because we don't know for sure that
we've included all of their contents in the new pack (not to mention
that somebody asked to keep them, and we don't know why; we should
respect that).

> What I'd expect it to do here is ignore the 'repack.writeBitmaps =
> true' value if there's a .keep that needs to be respected. Is this not
> a correct assumption?

In practice, I think that ends up worse. The .keep files are used by
receive-pack as lockfiles for incoming pushes. So imagine you kick off a
full repack just as somebody is pushing, and repack sees the temporary
.keep file. Your options are:

  1. Disable bitmaps, leaving the repository with no bitmaps at all
     until the next repack (because you're deleting the old bitmaps
     along with the old, non-kept pack).

  2. Duplicate the newly pushed objects in the pack (if they're even
     reachable; you're also racing to see the ref updates). Now you have
     bitmaps, but you're wasting a bit of space to store the racy push
     twice (and it goes away next time you repack).

If you're running a Git server which depends on bitmaps for good
performance, then (2) is much better. And that's the default.

If you want to override it, you can pass --no-pack-kept-objects, or set
repack.packKeptObjects to false.

I think the documentation for --pack-kept-objects could be a bit more
clear for this case. It doesn't mention the default value, nor that what
you really want with "-b" is probably "--no-pack-kept-objects".

-Peff