[BUG] Lazy fetch in partial clone fails with "not our ref" since 5946a564 (odb/source: make read_object_info() pluggable)

[BUG] Lazy fetch in partial clone fails with "not our ref" since 5946a564 (odb/source: make read_object_info() pluggable)

[Patrick Ohly]

Hello,

"git merge" in a blobless partial clone fails with "not our ref" while
lazily fetching a missing object. This started to occur today in the
Kubernetes CI, completely breaking the merging of PRs. We should have
been using git 2.54.0, the affected version, for a few days at least
without issues, so it can't be ruled out that this is at least partly
caused by a change on the GitHub side. We are about to revert to git
2.52.0 to unblock the Kubernetes project.

What follows is an agent analysis of the problem. I'm not rewriting it
because it seems succinct and useful as-is:

What did you do before the bug happened? (Steps to reproduce)

git init k && cd k
R=https://github.com/kubernetes/kubernetes.git
git fetch --filter=blob:none $R 6575724b63e50cd5dd454e774bc293c6de1e4e12
git checkout 6575724b63e50cd5dd454e774bc293c6de1e4e12
git fetch --filter=blob:none $R 787dd27af44c14c22d3bae46bf62b4a0271efc06
git -c user.name=a -c user.email=a@b merge --no-ff 787dd27af44c14c22d3bae46bf62b4a0271efc06

What did you expect to happen?

The merge succeeds, with missing blobs fetched lazily from the promisor remote.

What happened instead?

fatal: remote error: upload-pack: not our ref 27b64cc863187e62d8b47196a78c045412d32ac9
fatal: could not fetch 3c804b88bcf177a1435ba770fa9054c25e1315c3 from promisor remote

Bisect result

I bisected v2.52.0 (good) to v2.54.0 (bad). The first bad commit is:

5946a564cddc0bf471f27ae4c3fe205441e3ef65
odb/source: make `read_object_info()` function pluggable

The parent, 3bc3177ad7 ( 2.53.0.185.g3bc3177ad7 ), is good.

Versions tested

┌───────────────────────────────────────────────────────┬────────┐
│ Version                                               │ Result │
├───────────────────────────────────────────────────────┼────────┤
│ 2.39.5, 2.43.4, 2.45.2, 2.47.2, 2.49.1, 2.52.0        │ OK     │
├───────────────────────────────────────────────────────┼────────┤
│ 2.54.0, 2.56.0                                        │ fail   │
├───────────────────────────────────────────────────────┼────────┤
│ master (v2.56.0-341-gebdc087dc, built with NO_RUST=1) │ fail   │
└───────────────────────────────────────────────────────┴────────┘

Notes

• The remote is github.com over HTTPS. I haven't tried other hosts.
• I haven't analysed the cause. My guess is that the new  OBJECT_INFO_SECOND_READ  and reprepare path changes how the promisor lazy fetch is triggered. The client then requests an object by OID that the server refuses with "not our ref".
• The failure is deterministic. The first PR head merges fine on 2.52.0 and older.

[System Info]

git version 2.56.GIT (also 2.54.0 and 2.56.0), Linux x86_64 (Debian stable container)

-- 
Best Regards

Patrick Ohly
Cloud Software Architect

Re: [BUG] Lazy fetch in partial clone fails with "not our ref" since 5946a564 (odb/source: make read_object_info() pluggable)

[Kristofer Karlsson]

On Tue, 3 Nov 2026 at 14:31, Patrick Ohly <patrick.ohly@intel.com> wrote:
>
> Hello,
>
> "git merge" in a blobless partial clone fails with "not our ref" while
> lazily fetching a missing object. This started to occur today in the
> Kubernetes CI, completely breaking the merging of PRs. We should have
> been using git 2.54.0, the affected version, for a few days at least
> without issues, so it can't be ruled out that this is at least partly
> caused by a change on the GitHub side. We are about to revert to git
> 2.52.0 to unblock the Kubernetes project.
>
> [...]
>
> I bisected v2.52.0 (good) to v2.54.0 (bad). The first bad commit is:
>
> 5946a564cddc0bf471f27ae4c3fe205441e3ef65
> odb/source: make `read_object_info()` function pluggable
>
> The parent, 3bc3177ad7 ( 2.53.0.185.g3bc3177ad7 ), is good.

Thanks for the detailed report.  I think I have found the cause: I
can reproduce the failure with your steps and make it go away with a
small change to the loose object cache.

The problem is in the semantics of the loose object cache, which
answers quick existence checks.  It is a two-layer cache: a bitmap
keyed by the first byte of the object ID, recording which
"objects/XX/" subdirectories have been listed, and an oidtree with
the objects found there.  The intended invariant is that once a
subdirectory has been listed, the oidtree holds every loose object
in it, so that "not in the oidtree" means "does not exist".

Writing a loose object does not update the cache, which breaks the
invariant if the subdirectory had already been listed.  In
practice, that means a quick lookup of any object in the same
subdirectory earlier in the same process.

In your example, the object the server refuses, 27b64cc8, is not in
either parent nor in the merge base.  It is the content-merged blob
that the merge itself just wrote as a loose object.  merge-ort's
prefetch had already listed "27/", so the checkout of the merge
result thinks the blob is missing and asks the server for an object
that only exists locally.  The server rejects the whole batch, and
we then die on the first object in it that really is missing,
3c804b88, which is why the two OIDs in the error differ.

The bug is older than 5946a564cd (odb/source: make
`read_object_info()` function pluggable), but that commit, which
matches the bisect result, is what exposed it.  Before it, any
non-quick lookup that missed called odb_reprepare(), and that also
threw away the whole loose cache.  A merge does such lookups between
writing the merged blob and checking out the result, so in practice
the stale cache was discarded before anything could use it.  Since
that commit, the retry only reprepares the packfile store, and the
stale cache survives until the checkout.

I think the fix is to maintain the invariant: after a successful
loose object write, if the subdirectory has already been listed,
insert the new OID into the oidtree.  (Clearing the subdirectory's
bit would also work, at the cost of listing it again.)

I have a local fix with a unit test for the loose source.  An
end-to-end test is awkward, since it needs a merged blob whose first
byte matches one of the prefetched objects.

I could polish this into a proper patch, but since Patrick
Steinhardt has been reworking this part of the code, I think his
thoughts on the approach would be useful first.

Thanks,
Kristofer

Re: [BUG] Lazy fetch in partial clone fails with "not our ref" since 5946a564 (odb/source: make read_object_info() pluggable)

[Patrick Ohly]

Kristofer Karlsson <krka@spotify.com> writes:

> On Tue, 3 Nov 2026 at 14:31, Patrick Ohly <patrick.ohly@intel.com> wrote:
>>
>> Hello,
>>
>> "git merge" in a blobless partial clone fails with "not our ref" while
>> lazily fetching a missing object. This started to occur today in the
>> Kubernetes CI, completely breaking the merging of PRs. We should have
>> been using git 2.54.0, the affected version, for a few days at least
>> without issues, so it can't be ruled out that this is at least partly
>> caused by a change on the GitHub side. We are about to revert to git
>> 2.52.0 to unblock the Kubernetes project.
>>
>> [...]
>>
>> I bisected v2.52.0 (good) to v2.54.0 (bad). The first bad commit is:
>>
>> 5946a564cddc0bf471f27ae4c3fe205441e3ef65
>> odb/source: make `read_object_info()` function pluggable
>>
>> The parent, 3bc3177ad7 ( 2.53.0.185.g3bc3177ad7 ), is good.
>
> Thanks for the detailed report.  I think I have found the cause: I
> can reproduce the failure with your steps and make it go away with a
> small change to the loose object cache.

Any idea why this suddenly started to occur?

We were using 2.54.0 for several days without issues prior to the
failure. For example,
https://storage.googleapis.com/kubernetes-ci-logs/pr-logs/pull/batch/pull-kubernetes-cmd/2117551425255903232/clone-log.txt
merged several different PRs for
https://prow.k8s.io/view/gs/kubernetes-ci-logs/pr-logs/pull/batch/pull-kubernetes-cmd/2117551425255903232
using
us-docker.pkg.dev/k8s-infra-prow/images/clonerefs:v20261028-c06373a3e
as environment. It contains 2.54.0.

-- 
Best Regards

Patrick Ohly
Cloud Software Architect

Re: [BUG] Lazy fetch in partial clone fails with "not our ref" since 5946a564 (odb/source: make read_object_info() pluggable)

[Patrick Steinhardt]

On Wed, Nov 04, 2026 at 12:56:53PM +0100, Patrick Ohly wrote:
> Kristofer Karlsson <krka@spotify.com> writes:
> 
> > On Tue, 3 Nov 2026 at 14:31, Patrick Ohly <patrick.ohly@intel.com> wrote:
> >>
> >> Hello,
> >>
> >> "git merge" in a blobless partial clone fails with "not our ref" while
> >> lazily fetching a missing object. This started to occur today in the
> >> Kubernetes CI, completely breaking the merging of PRs. We should have
> >> been using git 2.54.0, the affected version, for a few days at least
> >> without issues, so it can't be ruled out that this is at least partly
> >> caused by a change on the GitHub side. We are about to revert to git
> >> 2.52.0 to unblock the Kubernetes project.
> >>
> >> [...]
> >>
> >> I bisected v2.52.0 (good) to v2.54.0 (bad). The first bad commit is:
> >>
> >> 5946a564cddc0bf471f27ae4c3fe205441e3ef65
> >> odb/source: make `read_object_info()` function pluggable
> >>
> >> The parent, 3bc3177ad7 ( 2.53.0.185.g3bc3177ad7 ), is good.
> >
> > Thanks for the detailed report.  I think I have found the cause: I
> > can reproduce the failure with your steps and make it go away with a
> > small change to the loose object cache.
> 
> Any idea why this suddenly started to occur?
> 
> We were using 2.54.0 for several days without issues prior to the
> failure. For example,
> https://storage.googleapis.com/kubernetes-ci-logs/pr-logs/pull/batch/pull-kubernetes-cmd/2117551425255903232/clone-log.txt
> merged several different PRs for
> https://prow.k8s.io/view/gs/kubernetes-ci-logs/pr-logs/pull/batch/pull-kubernetes-cmd/2117551425255903232
> using
> us-docker.pkg.dev/k8s-infra-prow/images/clonerefs:v20261028-c06373a3e
> as environment. It contains 2.54.0.

The issue here depends on the actual objects IDs that you're merging.
You need to merge at least one object with the same first two hex
characters into a newly written object that same first two characters.
If so, you can trigger a stale loose object cache that makes Git think
that the object does not exist locally, and it then tries to fetch it
from the promisor remote.

I've sent a fix for this bug via [1]. Thanks!

Patrick

[1]: <20261104-pks-odb-loose-stale-cache-v1-0-a00d5cdb5e7d@pks.im>

Re: [BUG] Lazy fetch in partial clone fails with "not our ref" since 5946a564 (odb/source: make read_object_info() pluggable)

[Patrick Steinhardt]

Hi,

On Tue, Nov 03, 2026 at 02:24:51PM +0100, Patrick Ohly wrote:
> Hello,
> 
> "git merge" in a blobless partial clone fails with "not our ref" while
> lazily fetching a missing object. This started to occur today in the
> Kubernetes CI, completely breaking the merging of PRs. We should have
> been using git 2.54.0, the affected version, for a few days at least
> without issues, so it can't be ruled out that this is at least partly
> caused by a change on the GitHub side. We are about to revert to git
> 2.52.0 to unblock the Kubernetes project.
> 
> What follows is an agent analysis of the problem. I'm not rewriting it
> because it seems succinct and useful as-is:
> 
> What did you do before the bug happened? (Steps to reproduce)
> 
> git init k && cd k
> R=https://github.com/kubernetes/kubernetes.git
> git fetch --filter=blob:none $R 6575724b63e50cd5dd454e774bc293c6de1e4e12
> git checkout 6575724b63e50cd5dd454e774bc293c6de1e4e12
> git fetch --filter=blob:none $R 787dd27af44c14c22d3bae46bf62b4a0271efc06
> git -c user.name=a -c user.email=a@b merge --no-ff 787dd27af44c14c22d3bae46bf62b4a0271efc06
> 
> What did you expect to happen?
> 
> The merge succeeds, with missing blobs fetched lazily from the promisor remote.
> 
> What happened instead?
> 
> fatal: remote error: upload-pack: not our ref 27b64cc863187e62d8b47196a78c045412d32ac9
> fatal: could not fetch 3c804b88bcf177a1435ba770fa9054c25e1315c3 from promisor remote

thanks for this well-written report, I can reproduce the bug.

> Bisect result
> 
> I bisected v2.52.0 (good) to v2.54.0 (bad). The first bad commit is:
> 
> 5946a564cddc0bf471f27ae4c3fe205441e3ef65
> odb/source: make `read_object_info()` function pluggable
> 
> The parent, 3bc3177ad7 ( 2.53.0.185.g3bc3177ad7 ), is good.

The root cause of this is actually older than the commit you have
bisected to: we never update the loose object cache when writing
objects, so in case we have already populated the specific sharding
directory that the newly written object belongs to the cache will be
stale. And as object existence checks are answered via the cache, we
will now pretend as if the object didn't exist at all.

The commit changes logic to not flush the loose object cache anymore in
case we cannot find an object. But the fact that flushing the loose
object cache fixes the bug was pure coincidence, as it required us to do
interleaving objects reads and object existence checks.

I've sent fixes via [1]. Thanks!

Patrick

[1]: <20261104-pks-odb-loose-stale-cache-v1-0-a00d5cdb5e7d@pks.im>