Fetching upstream remote fails if repo was a blobless clone

Fetching upstream remote fails if repo was a blobless clone

[Justin Su]

Thank you for filling out a Git bug report!
Please answer the following questions to help us understand your issue.

What did you do before the bug happened? (Steps to reproduce your issue)

```
git clone --filter=blob:none git@github.com:injust/delta
cd delta/
git remote add upstream git@github.com:dandavison/delta
git fetch upstream
```

What did you expect to happen? (Expected behavior) Fetch the upstream repo

What happened instead? (Actual behavior)

```
remote: Enumerating objects: 1578, done.
remote: Counting objects: 100% (776/776), done.
remote: Compressing objects: 100% (15/15), done.
remote: Total 1578 (delta 772), reused 761 (delta 761), pack-reused 802 (from 2)
Receiving objects: 100% (1578/1578), 2.67 MiB | 5.34 MiB/s, done.
Resolving deltas: 100% (1156/1156), completed with 356 local objects.
fatal: did not receive expected object 0020d54b979cc8cf59a13406f98bfe515b190559
fatal: fetch-pack: invalid index-pack output
```

Anything else you want to add: This only happens if a blobless clone
is performed


[System Info]
git version:
git version 2.50.1
cpu: x86_64
no commit associated with this build
sizeof-long: 8
sizeof-size_t: 8
shell-path: /bin/sh
feature: fsmonitor--daemon
libcurl: 8.6.0
zlib: 1.2.12
SHA-1: SHA1_DC
SHA-256: SHA256_BLK
uname: Darwin 24.5.0 Darwin Kernel Version 24.5.0: Tue Apr 22 19:53:26
PDT 2025; root:xnu-11417.121.6~2/RELEASE_X86_64 x86_64
compiler info: clang: 16.0.0 (clang-1600.0.26.6)
libc info: no libc information available
$SHELL (typically, interactive shell): /usr/local/bin/fish

Re: Fetching upstream remote fails if repo was a blobless clone

[Jeff King]

On Fri, Aug 01, 2025 at 05:30:56AM -0400, Justin Su wrote:

> git clone --filter=blob:none git@github.com:injust/delta
> cd delta/
> git remote add upstream git@github.com:dandavison/delta
> git fetch upstream
> [...]

> ```
> remote: Enumerating objects: 1578, done.
> remote: Counting objects: 100% (776/776), done.
> remote: Compressing objects: 100% (15/15), done.
> remote: Total 1578 (delta 772), reused 761 (delta 761), pack-reused 802 (from 2)
> Receiving objects: 100% (1578/1578), 2.67 MiB | 5.34 MiB/s, done.
> Resolving deltas: 100% (1156/1156), completed with 356 local objects.
> fatal: did not receive expected object 0020d54b979cc8cf59a13406f98bfe515b190559
> fatal: fetch-pack: invalid index-pack output
> ```

Hmm. I can't reproduce here, but I wonder if it is dependent on the
server-side repo state. E.g., if it depends on the fetch from the
upstream remote receiving a delta against a blob from origin. And that
might change if the server repacks, or even if there is another push.

Is it still happening for you?

-Peff

Re: Fetching upstream remote fails if repo was a blobless clone

[Justin Su]

On Sat, Aug 2, 2025 at 5:32 AM Jeff King <peff@peff.net> wrote:

> Is it still happening for you?

Yes, I can still reproduce with that sequence of commands.

Re: Fetching upstream remote fails if repo was a blobless clone

[Justin Su]

Turns out this was because I had `transfer.fsckObjects = true` in my
global config.

I think you should be able to repro if you change the last command to
`git -c fetch.fsckObjects=true fetch upstream`.


On Sat, Aug 2, 2025 at 2:02 PM Justin Su <injustsu@gmail.com> wrote:
>
> On Sat, Aug 2, 2025 at 5:32 AM Jeff King <peff@peff.net> wrote:
>
> > Is it still happening for you?
>
> Yes, I can still reproduce with that sequence of commands.

Re: Fetching upstream remote fails if repo was a blobless clone

[Jeff King]

On Sat, Aug 02, 2025 at 02:28:24PM -0400, Justin Su wrote:

> Turns out this was because I had `transfer.fsckObjects = true` in my
> global config.
> 
> I think you should be able to repro if you change the last command to
> `git -c fetch.fsckObjects=true fetch upstream`.

Thanks, I can reproduce easily now. The object in question isn't
mentioned directly in the pack at all, as an incoming object or as a
delta. It's mentioned by a tree, c5b8c11446. And then when we fsck, we
hit it via fsck_walk_tree(). And then when we've finished indexing the
pack, we check for any objects that were mentioned but which we don't
have. And we don't have 0020d54b979, so we barf.

I assume what's happening is that 0020d54b979 is contained in the origin
repo, but we don't fetch (because of the blob:none filter). And then
when we talk to the upstream repo, it assumes we _do_ have it because of
the commits that we claimed to have. And that looks like the case. In
the partial clone we can do:

  $ git rev-list --objects --all --missing=print-info | grep 0020d54b
  ?0020d54b979cc8cf59a13406f98bfe515b190559 path=src/features/navigate.rs type=blob

There it is, mentioned by the origin repo.

So it is perfectly normal for us to be missing this object, and
index-pack is wrong to complain. Curiously, there's this code in
fetch-pack.c:

                  if (args->from_promisor)
                          /*
                           * create_promisor_file() may be called afterwards but
                           * we still need index-pack to know that this is a
                           * promisor pack. For example, if transfer.fsckobjects
                           * is true, index-pack needs to know that .gitmodules
                           * is a promisor object (so that it won't complain if
                           * it is missing).
                           */
                          strvec_push(&cmd.args, "--promisor");

which you'd think would kick in here. And I confirmed that the
index-pack which barfs is passed that option.

So I dunno. Clearly there is a bug, but it's not clear to me how this
code is actually supposed to work.

Doing this:

diff --git a/builtin/index-pack.c b/builtin/index-pack.c
index 0a5c8a1ac8..e01cf7238b 100644
--- a/builtin/index-pack.c
+++ b/builtin/index-pack.c
@@ -262,9 +262,14 @@ static unsigned check_object(struct object *obj)
 		unsigned long size;
 		int type = odb_read_object_info(the_repository->objects,
 						&obj->oid, &size);
-		if (type <= 0)
+		if (type <= 0) {
+			if (is_promisor_object(the_repository, &obj->oid)) {
+				obj->flags |= FLAG_CHECKED;
+				return 1;
+			}
 			die(_("did not receive expected object %s"),
 			      oid_to_hex(&obj->oid));
+		}
 		if (type != obj->type)
 			die(_("object %s: expected type %s, found %s"),
 			    oid_to_hex(&obj->oid),

makes the problem go away. But I feel like I'm probably missing
something (and that function is rather expensive to run, though maybe
not so bad if the alternative is crashing).

+cc Jonathan Tan as the author of the code comment above for any wisdom.

-Peff

Re: Fetching upstream remote fails if repo was a blobless clone

[Jeff King]

On Sat, Aug 02, 2025 at 03:31:11PM -0400, Jeff King wrote:

> Doing this:
> 
> diff --git a/builtin/index-pack.c b/builtin/index-pack.c
> index 0a5c8a1ac8..e01cf7238b 100644
> --- a/builtin/index-pack.c
> +++ b/builtin/index-pack.c
> @@ -262,9 +262,14 @@ static unsigned check_object(struct object *obj)
>  		unsigned long size;
>  		int type = odb_read_object_info(the_repository->objects,
>  						&obj->oid, &size);
> -		if (type <= 0)
> +		if (type <= 0) {
> +			if (is_promisor_object(the_repository, &obj->oid)) {
> +				obj->flags |= FLAG_CHECKED;
> +				return 1;
> +			}
>  			die(_("did not receive expected object %s"),
>  			      oid_to_hex(&obj->oid));
> +		}
>  		if (type != obj->type)
>  			die(_("object %s: expected type %s, found %s"),
>  			    oid_to_hex(&obj->oid),
> 
> makes the problem go away. But I feel like I'm probably missing
> something (and that function is rather expensive to run, though maybe
> not so bad if the alternative is crashing).
> 
> +cc Jonathan Tan as the author of the code comment above for any wisdom.

And here is a minimal reproduction that doesn't depend on any other
repositories:

-- >8 --
# Server has two commits, with two blobs for file, old and new.
git init server
(
	cd server
	echo old >file
	git add .
	git commit -m old
	echo new >file
	git commit -am new

	git config uploadpack.allowfilter true
)

# The fork has built a new tree which mentions the old file.
git clone server fork
(
	cd fork
	git reset --hard HEAD^
	echo content >unrelated
	git add .
	git commit -m unrelated
)

# After our partial clone, we have the new blob (because we faulted it in to
# checkout), but not the old one (because it is buried in history).
git clone --no-local --filter=blob:none server repo
cd repo

# This will get the tree at the tip of the fork repo, which mentions old. When
# we fsck that tree, we'll see that it mentions the old blob, and expect to
# find it. But we won't due to the partial clone (though we could get it if we
# wanted from the server repo).
git -c transfer.fsckObjects=true fetch ../fork
-- >8 --

That fails with stock git now, like this:

  fatal: did not receive expected object 3367afdbbf91e638efe983616377c60477cc6612
  fatal: index-pack failed

but succeeds with the patch above.

-Peff