Re: Performance regression in connectivity check during receive-pack (git 2.54)

[Jeff King <peff@peff.net>]

On Tue, Jul 21, 2026 at 03:17:23PM +1200, Wolfgang Kritzinger wrote:

> `strace` shows that after a push, 2.54 does a failing open() of
> of numerous loose objects -- once in the quarantine (incoming)
> directory and once in the main object store -- before finding it
> in a pack:
> 
> openat(".../objects/tmp_objdir-incoming-XXXX/ed/58..", O_RDONLY) = ENOENT
> openat(".../objects/ed/58..", O_RDONLY) = ENOENT

Interesting. Here's a smaller reproduction recipe that shows the issue:

  # clone of git.git, or any other non-trivial repo; it should be mostly
  # packed
  src=/path/to/git

  git init empty
  export GIT_ALTERNATE_OBJECT_DIRECTORIES=$src/.git/objects
  strace -fe openat \
    git -C empty rev-list --objects $(git -C $src rev-parse HEAD) >/dev/null

In v2.50, we see almost no loose object open calls, because we check the
pack first. But in v2.54, we see tons of them.

> 2.50 does not do this. In most customer deployments of Bitbucket,
> the Git data lives on an NFS share. The extra latency on NFS makes
> this process of checking for non-existent loose objects take too
> long, the push essentially hangs at the "Checking connectivity" step.

Yeah, I can imagine. But even on a fast filesystem, we definitely want
to avoid all of those syscalls. Replacing "strace" above with a timing
harness, even on a system with fast syscalls and a warm cache, the v2.54
version is ~12% slower.

> I believe this new behavior was introduced in the recent object
> database rework. After using bisect, I belive the problem can be
> traced back to commit 8384cbcb4c.

Hmm, my bisect ended up at a593373b09 (packfile: refactor
`find_pack_entry()` to work on the packfile store, 2026-01-09), which is
nearby. I'm not sure if it might depend on other factors (e.g., presence
of commit graphs, midx, etc) or my reproduction is not exactly like
yours, or if one of us messed up bisection.

+cc Patrick as the author of both commits.

> I don't know the codebase well, but from what I can see is that
> the order in which objects are looked up in object databases
> changed.
> 
> Assuming there are two object databases configured (Main repo,
> and the quarantine directory, for example), the lookup order used
> to be:
> 
> 1. _quarantine dir_ packs
> 2. _main dir_ packs
> 3. _quarantine dir_ loose objects
> 4. _main dir_ loose objects
> 
> With Git 2.54, the order appears to have changed to:
> 
> 1. _quarantine dir_ packs
> 2. _quarantine dir_ loose objects
> 3. _main dir_ packs
> 4. _main dir_ loose objects
> 
> In my testing, within a well-packed repo, Git 2.50 actually never
> performed a loose object lookup.

Yeah, and that type of regression makes sense for what a593373b09 was
trying to do. But I think the v2.54 behavior is wrong. We should check
all packs before any loose objects.

I'm not sure of the correct fix. This is working against the whole "odb
sources are independent and abstract" refactoring that a593373b09 was
going for. But I think it's an important optimization. I guess the
abstract version would be that each source has "fast" and "slow" lookups
or something like that, and we check all fast ones before slow ones. But
that is pretty gross.

I'll leave it to Patrick to ponder further. I haven't really been paying
a lot of attention to the odb refactoring.

-Peff