Git development
 help / color / mirror / Atom feed
From: Jeff King <peff@peff.net>
To: Michael Montalbo <mmontalbo@gmail.com>
Cc: Patrick Steinhardt <ps@pks.im>,
	git@vger.kernel.org, Junio C Hamano <gitster@pobox.com>
Subject: [PATCH 3/3] t5551: pack refs after creating many tags
Date: Sun, 28 Jun 2026 04:07:10 -0400	[thread overview]
Message-ID: <20260628080710.GC107826@coredump.intra.peff.net> (raw)
In-Reply-To: <20260628075716.GA3525066@coredump.intra.peff.net>

We have two tests that create 2,000 and 100,000 tags respectively.
After doing so, the resulting state can be a bit slow to work with when
using the "files" ref backend, as each of those refs is in its own file.

This isn't a very realistic scenario, as we'd expect most of those refs
to be packed. If they accrue over time along with objects, they'd get
packed by maintenance/gc runs. And if you have a process that creates a
ton of refs at once (like a big fast-import), the usual recommendation
is to run maintenance afterwards.

So let's follow that recommendation and pack the refs ourselves.
Unfortunately, this does not seem to produce an improvement to the
run-time of the test script! That's because after producing this state,
we perform only a few fetches of it. And packing the refs costs at least
as much as serving a ref advertisement (both have to iterate the refs,
but packing additionally must write .lock files as we pack).

My wall-clock time was slightly improved (but within the noise) with
this patch, but my user and system CPU time were slightly worse!
However, on a loaded system with I/O bottlenecks, it may be a net win.
That's somewhat of a guess, though.

It would be nice if we had a way to generate all of these refs without
writing so many individual files. But even if we taught the ref code to
write large cases directly to the packed-refs file, we'd still need to
take individual locks. The real solution is a backend like reftable,
which shaves ~30% off of the test runtime.

Signed-off-by: Jeff King <peff@peff.net>
---
I'm iffy on whether this one is worth it.

If you apply just this patch without patch 2, then the run-time does
improve quite a bit. The cost of packing is amortized by the improved
performance for all of those subsequent tests (but after patch 2, they
never even see the unpacked state).

Likewise, I suspect this would make our timeout problems go away even
without patch 1.

So the whole series _could_ be reduced to just this one patch. But
hopefully the reasoning given in the earlier patches makes sense, at
which point this one is kind of superfluous.

 t/t5551-http-fetch-smart.sh | 1 +
 1 file changed, 1 insertion(+)

diff --git a/t/t5551-http-fetch-smart.sh b/t/t5551-http-fetch-smart.sh
index cd851f24b8..e2e729216f 100755
--- a/t/t5551-http-fetch-smart.sh
+++ b/t/t5551-http-fetch-smart.sh
@@ -393,6 +393,7 @@ create_tags () {
 	tag=$(perl -e "print \"bla\" x 30") &&
 	sed -e "s|^:\([^ ]*\) \(.*\)$|create refs/tags/$tag-\1 \2|" <marks >input &&
 	git update-ref --stdin <input &&
+	git pack-refs --all &&
 	rm input
 }
 
-- 
2.55.0.rc2.353.gf769b6597e

  parent reply	other threads:[~2026-06-28  8:07 UTC|newest]

Thread overview: 20+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-06-20 15:33 [RFH] Why do osx CI jobs so unreliable? Michael Montalbo
2026-06-21 21:34 ` Jeff King
2026-06-22  4:42   ` Patrick Steinhardt
2026-06-22  9:47     ` Patrick Steinhardt
2026-06-22  9:55       ` Patrick Steinhardt
2026-06-22 10:29         ` Patrick Steinhardt
2026-06-26  3:27           ` Michael Montalbo
2026-06-26  5:16             ` Jeff King
2026-06-26 10:50               ` Patrick Steinhardt
2026-06-26 13:45                 ` Junio C Hamano
2026-06-26 23:26                 ` Michael Montalbo
2026-06-28  7:57                   ` [PATCH 0/3] fixing expensive http test timeouts Jeff King
2026-06-28  8:00                     ` [PATCH 1/3] t/lib-httpd: bump apache timeout Jeff King
2026-06-28  8:03                     ` [PATCH 2/3] t5551: put many-tags case into its own repo Jeff King
2026-06-28 21:44                       ` Junio C Hamano
2026-06-29  0:34                         ` Jeff King
2026-06-28  8:07                     ` Jeff King [this message]
2026-06-28 21:25                       ` [PATCH 3/3] t5551: pack refs after creating many tags Junio C Hamano
2026-06-26 23:43                 ` [RFH] Why do osx CI jobs so unreliable? Jeff King
2026-06-22  5:05   ` Junio C Hamano

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20260628080710.GC107826@coredump.intra.peff.net \
    --to=peff@peff.net \
    --cc=git@vger.kernel.org \
    --cc=gitster@pobox.com \
    --cc=mmontalbo@gmail.com \
    --cc=ps@pks.im \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox