Re: slow object packing during push

["Ævar Arnfjörð Bjarmason" <avarab@gmail.com>]

On Tue, Mar 09 2021, David Turner wrote:

> I have a large, funny repository that has very slow pushes unless I
> set pack.usebitmaps=false to false.

Good to see you on-list again! :)

> First, a description of the repo: it's about 175GB, and was created by combining about 40,000 smaller repositories.  Historically, these repos were submodules of one meta repository[2].  I have stitched together the submodules, and this is the repository in which the stitching was done - that is, it contains all of the objects from the smaller repos, plus all of the objects from the meta repository, plus the newly-created trees & commits for the stitched repositories.  As new commits come into the meta repository (which have gitlinks to new submodule commits), we fetch from the meta repository (8s - it would be 2s if we were fetching into a normal clone without all of the other stuff), and the submodules (up to 10s per and embarrassingly parallel). Then we stitch (~0s), and push to the stitched "unity" repository (~2 minutes!!!).  The entire repo fits in RAM (yes, all 175G) and is in fact in the disk cache (I prewarmed the cache before testing anything).  
>
> The vast majority of the time appears to be spent in git pack-objects, and in particular in the stack trace in [1].  If I set pack.usebitmaps=false, the push only takes 10s.   This seems like pack bitmaps are a severe pessimization for my purposes.  This is true even immediately after a repack (that is, almost all of the objects are in one giant pack, except the newly-fetched ones).  I also tried setting up pack islands - one for each smaller repo, one for the stitched commits, and one for commits from the meta repo.  I'm not sure if this is necessary, but it's definitely not sufficient (my current config has it turned on, because I didn't feel like repacking again after testing it, and I tested it before testing pack.usebimaps). 
>
>
> [1]
> #9  0x000055d849183bfe in traverse_trees_and_blobs (ctx=ctx@entry=0x7fff2de42a80, 
>     base=base@entry=0x7fff2de42a30) at list-objects.c:344
> #10 0x000055d849183d2b in do_traverse (ctx=ctx@entry=0x7fff2de42a80) at list-objects.c:388
> #11 0x000055d84918408f in traverse_commit_list_filtered (
>     filter_options=filter_options@entry=0x55d849544e80 <filter_options>, 
>     revs=revs@entry=0x7fff2de43f00, show_commit=show_commit@entry=0x55d8491a7830 <show_commit>, 
>     show_object=show_object@entry=0x55d8491ac920 <show_object>, 
>     show_data=show_data@entry=0x7fff2de42b50, omitted=omitted@entry=0x0) at list-objects.c:421
> #12 0x000055d8491a8c1a in find_objects (bitmap_git=bitmap_git@entry=0x55d84a41cd40, 
>     revs=revs@entry=0x7fff2de43f00, roots=0x0, seen=seen@entry=0x0, 
>     filter=filter@entry=0x55d849544e80 <filter_options>) at pack-bitmap.c:603
> #13 0x000055d8491af68d in prepare_bitmap_walk (revs=revs@entry=0x7fff2de43f00, 
>     filter=filter@entry=0x55d849544e80 <filter_options>) at pack-bitmap.c:1004
> #14 0x000055d8490b1983 in get_object_list_from_bitmap (revs=0x7fff2de43f00)
>     at builtin/pack-objects.c:3294
> #15 get_object_list (av=<optimized out>, ac=<optimized out>) at builtin/pack-objects.c:3373
> #16 cmd_pack_objects (argc=<optimized out>, argv=<optimized out>, prefix=<optimized out>)
>     at builtin/pack-objects.c:3739
> #17 0x000055d84903ed19 in run_builtin (argv=<optimized out>, argc=<optimized out>, 
>     p=<optimized out>) at git.c:450
> #18 handle_builtin (argc=7, argv=0x7fff2de45320) at git.c:700
> #19 0x000055d84903fd96 in run_argv (argv=0x7fff2de450a0, argcp=0x7fff2de450ac) at git.c:767
> #20 cmd_main (argc=<optimized out>, argv=<optimized out>) at git.c:898
> #21 0x000055d84903e8ef in main (argc=8, argv=0x7fff2de45318) at common-main.c:52
>
> [2] https://github.com/twosigma/git-meta

Without having carefully re-read it, I believe this issue is the same as
what I reported here in 2019, and I think you'll find the resulting
discussion intresting:
https://lore.kernel.org/git/87zhoz8b9o.fsf@evledraar.gmail.com/

Having skimmed it, I think you're probably omitting that this is a bare
repo you're pushing from, and thus you're running into the combination
of repack.writeBitmaps being true by default on bare repos, and
pack.useBitmaps being true everywhere (but having no effect by default
unless you have a bare repo, unless you manually make bitmaps blah
blah).

One of the semi-conclusions from the above thread was that we mostly
turned this on thinking that bare ~= server that accepts pushes, and
bitmaps are known to be worse (but not always!) for when you're pushing
*from* your repo.