git clone with --dissociate sometimes fails to check out target commit

git clone with --dissociate sometimes fails to check out target commit

[Rasmus Villemoes]

Hi

We have now seen this error a couple of times in our CI, and this time I
managed to grab a snapshot of the local mirror for which it fails. The
failing command is

  git clone --verbose --depth=20 --branch=whinlatter --reference-if-able=/yocto/meta-mirrors/core --dissociate https://git.openembedded.org/openembedded-core core
  Cloning into 'core'...
  POST git-upload-pack (388 bytes)
  POST git-upload-pack (986 bytes)
  POST git-upload-pack (gzip 1836 to 958 bytes)
  fatal: unable to parse commit 8751ec83421192fc0f8495fb95798f9eb7be77a0
  warning: Clone succeeded, but checkout failed.
  You can inspect what was checked out with 'git status'
  and retry with 'git restore --source=HEAD :/'

I wrapped up that local copy /yocto/meta-mirrors/core in a tarball, but
it's ~200M, and I don't know another way of reproducing. I also don't
have a better way of sharing such a file than [1], apologies.

Using that repository as both the remote url to clone and the local
reference, I can consistently reproduce the problem. That is:

  cd /tmp
  # fetch that core.tar.gz
  mkdir upstream-core local-core
  tar -xf core.tar.gz -C upstream-core/
  tar -xf core.tar.gz -C local-core/
  git clone --verbose --branch=whinlatter --reference-if-able=/tmp/local-core --dissociate --depth=20 file:///tmp/upstream-core core

fails in the same way, with both git 2.47.3 (Debian trixie) and 2.53.0
(Arch). Removing --depth=20 doesn't change anything, neither does
removing --branch=whinlatter (except of course for the commit it tries
to check out). But dropping --dissociate, the clone works as expected.

It doesn't happen very often, the last time was around January 30, where
it was for another repository
(https://github.com/openembedded/meta-openembedded.git), but exactly the
same symptoms, so about 100 nightly pipelines ago.

Are we using --dissociate wrongly, or are we perhaps not maintaining
those local mirror repos properly? They are essentially just created
with 'git clone --mirror', with 'git remote update' run periodically.

Naively, I'd expect the effects of --dissociate to only happen after
everything else the clone command does has been done, but it seems that
the ties to the reference repo are cut too soon.

Rasmus

[1] https://prevasonline-my.sharepoint.com/:u:/g/personal/rasmus_villemoes_prevas_dk/IQCRaxpwj5NfQYZNQJWc9PJTAY0C33XvXn8CnqPEdPAbpDA?e=zQAfg7

Re: git clone with --dissociate sometimes fails to check out target commit

[Jeff King]

On Mon, May 04, 2026 at 10:20:32AM +0200, Rasmus Villemoes wrote:

> Are we using --dissociate wrongly, or are we perhaps not maintaining
> those local mirror repos properly? They are essentially just created
> with 'git clone --mirror', with 'git remote update' run periodically.
> 
> Naively, I'd expect the effects of --dissociate to only happen after
> everything else the clone command does has been done, but it seems that
> the ties to the reference repo are cut too soon.

No, you're using it correctly. The dissociate step should copy all of
the shared objects into the new repo, so it shouldn't matter whether we
do it before or after checkout. The objects are there either way.

But there's an interesting bug here with commit graphs. What happens is
this:

  1. During the initial part of the clone, we may load a commit object
     using the commit-graph file from the --reference repo. The commit
     struct is left with a blank "maybe_tree" field, because we know we
     can load it from the commit graph later (and don't want to spend
     the effort to make a "struct tree" unless somebody asks for it).

  2. During the dissociate step, we call odb_close(), since we're
     throwing away the link to the reference repo, and we don't want to
     use our in-process structs that point to it. That step also throws
     away our open reference to the commit-graph file, and the
     in-process slab that holds the graph positions we've loaded.

  3. The checkout process needs the tree, so it calls
     repo_get_commit_tree(). That sees that maybe_commit is NULL, so we
     check whether it might be loaded from the graph file. But when we
     ask about the graph position, we don't have one! It was in the slab
     we threw away. So we return NULL, and the caller thinks the commit
     is corrupt.

This is a bug that we theorized existed in a thread a while ago:

  https://lore.kernel.org/git/20240110113914.GE16674@coredump.intra.peff.net/

but we didn't have a way to trigger it. Now we do. Hooray, I guess? ;)

The fallback load suggested in that message fixes it (modulo the fact
that it forgot to return commit->maybe_tree at the end of the function).
Below is a slightly safer version of the same concept that likewise
fixes the problem.

It's kind of ugly, but I think may be the least-bad solution. See that
earlier thread for more discussion of alternatives.

In the meantime, doing your dissociate clone with:

  git -c core.commitGraph=false clone ...

should work around the problem.

---
diff --git a/commit.c b/commit.c
index 80d8d07875..50d736b339 100644
--- a/commit.c
+++ b/commit.c
@@ -434,16 +434,46 @@ static inline void set_commit_tree(struct commit *c, struct tree *t)
 	c->maybe_tree = t;
 }
 
+static void load_tree_from_commit_contents(struct repository *r, struct commit *commit)
+{
+	enum object_type type;
+	unsigned long size;
+	char *buf;
+	const char *p;
+	struct object_id tree_oid;
+
+	buf = odb_read_object(r->objects, &commit->object.oid, &type, &size);
+	if (!buf)
+		return;
+
+	if (type == OBJ_COMMIT &&
+	    skip_prefix(buf, "tree ", &p) &&
+	    !parse_oid_hex(p, &tree_oid, &p) &&
+	    *p == '\n')
+		commit->maybe_tree = lookup_tree(r, &tree_oid);
+
+	free(buf);
+}
+
 struct tree *repo_get_commit_tree(struct repository *r,
-				  const struct commit *commit)
+				  struct commit *commit)
 {
 	if (commit->maybe_tree || !commit->object.parsed)
 		return commit->maybe_tree;
 
 	if (commit_graph_position(commit) != COMMIT_NOT_FROM_GRAPH)
 		return get_commit_tree_in_graph(r, commit);
 
-	return NULL;
+	/*
+	 * This is either a corrupt commit, or one which we partially loaded
+	 * from a graph file but then subsequently threw away the graph data.
+	 *
+	 * Optimistically assume it's the latter and try to reload from
+	 * scratch. This gives a performance penalty if it really is a corrupt
+	 * commit, but presumably that happens rarely.
+	 */
+	load_tree_from_commit_contents(r, commit);
+	return commit->maybe_tree;
 }
 
 struct object_id *get_commit_tree_oid(const struct commit *commit)
diff --git a/commit.h b/commit.h
index 58150045af..5eb1264077 100644
--- a/commit.h
+++ b/commit.h
@@ -163,7 +163,7 @@ void repo_unuse_commit_buffer(struct repository *r,
  */
 void free_commit_buffer(struct parsed_object_pool *pool, struct commit *);
 
-struct tree *repo_get_commit_tree(struct repository *, const struct commit *);
+struct tree *repo_get_commit_tree(struct repository *, struct commit *);
 struct object_id *get_commit_tree_oid(const struct commit *);
 
 /*

Re: git clone with --dissociate sometimes fails to check out target commit

[Jeff King]

On Mon, May 04, 2026 at 05:51:10AM -0400, Jeff King wrote:

> No, you're using it correctly. The dissociate step should copy all of
> the shared objects into the new repo, so it shouldn't matter whether we
> do it before or after checkout. The objects are there either way.
> 
> But there's an interesting bug here with commit graphs. What happens is
> this:

Oh, and ironically dissociating later _would_ fix this bug, like so:

diff --git a/builtin/clone.c b/builtin/clone.c
index fba3c9c508..7b7c83c717 100644
--- a/builtin/clone.c
+++ b/builtin/clone.c
@@ -1616,11 +1616,6 @@ int cmd_clone(int argc,
 	transport_unlock_pack(transport, 0);
 	transport_disconnect(transport);
 
-	if (option_dissociate) {
-		odb_close(the_repository->objects);
-		dissociate_from_references();
-	}
-
 	if (option_sparse_checkout && git_sparse_checkout_init(dir))
 		return 1;
 
@@ -1630,6 +1625,11 @@ int cmd_clone(int argc,
 		       filter_submodules,
 		       ref_storage_format);
 
+	if (option_dissociate) {
+		odb_close(the_repository->objects);
+		dissociate_from_references();
+	}
+
 	list_objects_filter_release(&filter_options);
 
 	string_list_clear(&option_not, 0);


But only because we are working around it: if we dissociate at the very
end, then there is no in-process code that will look at the objects
after that odb_close() call, and thus the bug cannot be triggered. It
would still potentially be lurking for other odb_close() callers,
though.

-Peff

Re: git clone with --dissociate sometimes fails to check out target commit

[Rasmus Villemoes]

On Mon, May 04 2026, Jeff King <peff@peff.net> wrote:

> On Mon, May 04, 2026 at 10:20:32AM +0200, Rasmus Villemoes wrote:
>
>> Are we using --dissociate wrongly, or are we perhaps not maintaining
>> those local mirror repos properly? They are essentially just created
>> with 'git clone --mirror', with 'git remote update' run periodically.
>> 
>> Naively, I'd expect the effects of --dissociate to only happen after
>> everything else the clone command does has been done, but it seems that
>> the ties to the reference repo are cut too soon.
>
> No, you're using it correctly. The dissociate step should copy all of
> the shared objects into the new repo, so it shouldn't matter whether we
> do it before or after checkout. The objects are there either way.
>
[snip]
>
> It's kind of ugly, but I think may be the least-bad solution. See that
> earlier thread for more discussion of alternatives.
>
> In the meantime, doing your dissociate clone with:
>
>   git -c core.commitGraph=false clone ...
>
> should work around the problem.

Thanks for the extremely fast reply, analysis, patch and workaround!

I can confirm that the commit graph disabling workaround works on both
the Debian and Arch machines.

I can also confirm that the patch applied on top of v2.54.0 works,
although the build does throw this warning:

commit.c: In function ‘get_commit_tree_oid’:                                                                                                                                                                          
commit.c:481:66: warning: passing argument 2 of ‘repo_get_commit_tree’ discards ‘const’ qualifier from pointer target type [-Wdiscarded-qualifiers]                                                                   
  481 |         struct tree *tree = repo_get_commit_tree(the_repository, commit);                                                                                                                                     
      |                                                                  ^~~~~~                                                                                                                                       
commit.c:459:50: note: expected ‘struct commit *’ but argument is of type ‘const struct commit *’                                                                                                                     
  459 |                                   struct commit *commit)                                                                                                                                                      
      |                                   ~~~~~~~~~~~~~~~^~~~~~                                                          

Thanks again,
Rasmus