* [PATCH 0/3] compatObjectFormat: fix a BUG(), wrong id and unknown headers
@ 2026-11-12 19:57 Adrian Ratiu
2026-11-12 19:57 ` [PATCH 1/3] setup: reject compatObjectFormat == objectFormat Adrian Ratiu
` (2 more replies)
0 siblings, 3 replies; 10+ messages in thread
From: Adrian Ratiu @ 2026-11-12 19:57 UTC (permalink / raw)
To: git
Cc: Junio C Hamano, Patrick Steinhardt, brian m . carlson,
Eric W . Biederman, Axel Rasmussen, Emily Shaffer, Josh Steadmon,
Adrian Ratiu
Hello all,
I've been continuing my three-way Git, Gitoxide and Jujutsu sha1-sha256
interop testing and uncovered three more bugs, this time in Git's
compatObjectFormat.
I'm reasonably confident in the first two patches.
The last patch needs careful review, because there are two conflicting
behaviors: Git copies unknown commit headers when it creates commits,
but rejects them when converting existing ones.
I made both copy them, which removes a check added on purpose in
08a45903cb (object-file-convert: convert commits that embed signed tags,
2023-10-01) and I'm not 100% sure it's the right direction.
More details in the commit messages.
Many thanks,
Adrian
Adrian Ratiu (3):
setup: reject compatObjectFormat == objectFormat
rev-parse: die when --output-object-format cannot map an object
object-file-convert: copy unknown commit headers unchanged
builtin/rev-parse.c | 8 +++++---
object-file-convert.c | 21 +++++++++++----------
setup.c | 7 +++++++
t/t1016-compatObjectFormat.sh | 18 ++++++++++++++++++
t/t1302-repo-version.sh | 14 ++++++++++++++
t/t1500-rev-parse.sh | 18 ++++++++++++++++++
6 files changed, 73 insertions(+), 13 deletions(-)
--
2.56.1
^ permalink raw reply [flat|nested] 10+ messages in thread* [PATCH 1/3] setup: reject compatObjectFormat == objectFormat 2026-11-12 19:57 [PATCH 0/3] compatObjectFormat: fix a BUG(), wrong id and unknown headers Adrian Ratiu @ 2026-11-12 19:57 ` Adrian Ratiu 2026-11-16 7:36 ` Patrick Steinhardt 2026-11-12 19:57 ` [PATCH 2/3] rev-parse: die when --output-object-format cannot map an object Adrian Ratiu 2026-11-12 19:57 ` [PATCH 3/3] object-file-convert: copy unknown commit headers unchanged Adrian Ratiu 2 siblings, 1 reply; 10+ messages in thread From: Adrian Ratiu @ 2026-11-12 19:57 UTC (permalink / raw) To: git Cc: Junio C Hamano, Patrick Steinhardt, brian m . carlson, Eric W . Biederman, Axel Rasmussen, Emily Shaffer, Josh Steadmon, Adrian Ratiu When extensions.compatObjectFormat sets the same hash as the repo object format, every command in a build with Rust aborts with: BUG: repository.c:204: hash_algo and compat_hash_algo match This includes "git config", so the bad value can't be removed with Git. It also happens with no extensions.objectFormat at all, when compatObjectFormat is set to the default object format. Check for this in verify_repository_format(), next to the other repo format checks, and return an error instead. The check runs before any of the Rust-only code. Builds without Rust, which so far died because compatObjectFormat requires Rust, now get the same error, so the test needs no RUST prereq. Signed-off-by: Adrian Ratiu <adrian.ratiu@collabora.com> --- setup.c | 7 +++++++ t/t1302-repo-version.sh | 14 ++++++++++++++ 2 files changed, 21 insertions(+) diff --git a/setup.c b/setup.c index 186815c780..52a9462edd 100644 --- a/setup.c +++ b/setup.c @@ -918,6 +918,13 @@ int verify_repository_format(const struct repository_format *format, return -1; } + if (format->compat_hash_algo == format->hash_algo) { + strbuf_addf(err, _("'%s' is '%s', the same as the object format"), + "extensions.compatobjectformat", + hash_algos[format->hash_algo].name); + return -1; + } + return 0; } diff --git a/t/t1302-repo-version.sh b/t/t1302-repo-version.sh index 69723b88ff..858aea62a5 100755 --- a/t/t1302-repo-version.sh +++ b/t/t1302-repo-version.sh @@ -98,6 +98,20 @@ abort 0 noop-v1 allow 1 noop-v1 EOF +for hash in sha1 sha256 +do + test_expect_success "compatObjectFormat equal to objectFormat $hash" " + test_when_finished 'rm -rf same' && + git init --object-format=$hash same && + git -C same config core.repositoryformatversion 1 && + git -C same config extensions.compatObjectFormat $hash && + test_must_fail git -C same rev-parse --git-dir 2>err && + test_grep \"'extensions.compatobjectformat' is '$hash', the same as the object format\" err && + git config --file=same/.git/config --unset extensions.compatObjectFormat && + git -C same rev-parse --git-dir + " +done + test_expect_success 'precious-objects allowed' ' git config core.repositoryFormatVersion 1 && git config extensions.preciousObjects 1 && -- 2.56.1 ^ permalink raw reply related [flat|nested] 10+ messages in thread
* Re: [PATCH 1/3] setup: reject compatObjectFormat == objectFormat 2026-11-12 19:57 ` [PATCH 1/3] setup: reject compatObjectFormat == objectFormat Adrian Ratiu @ 2026-11-16 7:36 ` Patrick Steinhardt 2026-11-17 10:42 ` Adrian Ratiu 0 siblings, 1 reply; 10+ messages in thread From: Patrick Steinhardt @ 2026-11-16 7:36 UTC (permalink / raw) To: Adrian Ratiu Cc: git, Junio C Hamano, brian m . carlson, Eric W . Biederman, Axel Rasmussen, Emily Shaffer, Josh Steadmon On Thu, Nov 12, 2026 at 09:57:38PM +0200, Adrian Ratiu wrote: > diff --git a/setup.c b/setup.c > index 186815c780..52a9462edd 100644 > --- a/setup.c > +++ b/setup.c > @@ -918,6 +918,13 @@ int verify_repository_format(const struct repository_format *format, > return -1; > } > > + if (format->compat_hash_algo == format->hash_algo) { > + strbuf_addf(err, _("'%s' is '%s', the same as the object format"), > + "extensions.compatobjectformat", We should rather say "extensions.compatObjectFormat" here with proper casing, as this is user-facing. > + hash_algos[format->hash_algo].name); > + return -1; > + } > + > return 0; > } > > diff --git a/t/t1302-repo-version.sh b/t/t1302-repo-version.sh > index 69723b88ff..858aea62a5 100755 > --- a/t/t1302-repo-version.sh > +++ b/t/t1302-repo-version.sh > @@ -98,6 +98,20 @@ abort 0 noop-v1 > allow 1 noop-v1 > EOF > > +for hash in sha1 sha256 > +do > + test_expect_success "compatObjectFormat equal to objectFormat $hash" " > + test_when_finished 'rm -rf same' && > + git init --object-format=$hash same && > + git -C same config core.repositoryformatversion 1 && > + git -C same config extensions.compatObjectFormat $hash && > + test_must_fail git -C same rev-parse --git-dir 2>err && > + test_grep \"'extensions.compatobjectformat' is '$hash', the same as the object format\" err && > + git config --file=same/.git/config --unset extensions.compatObjectFormat && Nit: do we maybe want to use the modern alternate `git config unset`? If so, we should probably be consistent and also use `gi tconfig set` further up. Patrick ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [PATCH 1/3] setup: reject compatObjectFormat == objectFormat 2026-11-16 7:36 ` Patrick Steinhardt @ 2026-11-17 10:42 ` Adrian Ratiu 0 siblings, 0 replies; 10+ messages in thread From: Adrian Ratiu @ 2026-11-17 10:42 UTC (permalink / raw) To: Patrick Steinhardt Cc: git, Junio C Hamano, brian m . carlson, Eric W . Biederman, Axel Rasmussen, Emily Shaffer, Josh Steadmon On Mon, 16 Nov 2026, Patrick Steinhardt <ps@pks.im> wrote: > On Thu, Nov 12, 2026 at 09:57:38PM +0200, Adrian Ratiu wrote: >> diff --git a/setup.c b/setup.c >> index 186815c780..52a9462edd 100644 >> --- a/setup.c >> +++ b/setup.c >> @@ -918,6 +918,13 @@ int verify_repository_format(const struct repository_format *format, >> return -1; >> } >> >> + if (format->compat_hash_algo == format->hash_algo) { >> + strbuf_addf(err, _("'%s' is '%s', the same as the object format"), >> + "extensions.compatobjectformat", > > We should rather say "extensions.compatObjectFormat" here with proper > casing, as this is user-facing. Ack, will do in v2. >> + hash_algos[format->hash_algo].name); >> + return -1; >> + } >> + >> return 0; >> } >> >> diff --git a/t/t1302-repo-version.sh b/t/t1302-repo-version.sh >> index 69723b88ff..858aea62a5 100755 >> --- a/t/t1302-repo-version.sh >> +++ b/t/t1302-repo-version.sh >> @@ -98,6 +98,20 @@ abort 0 noop-v1 >> allow 1 noop-v1 >> EOF >> >> +for hash in sha1 sha256 >> +do >> + test_expect_success "compatObjectFormat equal to objectFormat $hash" " >> + test_when_finished 'rm -rf same' && >> + git init --object-format=$hash same && >> + git -C same config core.repositoryformatversion 1 && >> + git -C same config extensions.compatObjectFormat $hash && >> + test_must_fail git -C same rev-parse --git-dir 2>err && >> + test_grep \"'extensions.compatobjectformat' is '$hash', the same as the object format\" err && >> + git config --file=same/.git/config --unset extensions.compatObjectFormat && > > Nit: do we maybe want to use the modern alternate `git config unset`? If > so, we should probably be consistent and also use `gi tconfig set` > further up. Ack, will do in v2. ^ permalink raw reply [flat|nested] 10+ messages in thread
* [PATCH 2/3] rev-parse: die when --output-object-format cannot map an object 2026-11-12 19:57 [PATCH 0/3] compatObjectFormat: fix a BUG(), wrong id and unknown headers Adrian Ratiu 2026-11-12 19:57 ` [PATCH 1/3] setup: reject compatObjectFormat == objectFormat Adrian Ratiu @ 2026-11-12 19:57 ` Adrian Ratiu 2026-11-16 7:37 ` Patrick Steinhardt 2026-11-12 19:57 ` [PATCH 3/3] object-file-convert: copy unknown commit headers unchanged Adrian Ratiu 2 siblings, 1 reply; 10+ messages in thread From: Adrian Ratiu @ 2026-11-12 19:57 UTC (permalink / raw) To: git Cc: Junio C Hamano, Patrick Steinhardt, brian m . carlson, Eric W . Biederman, Axel Rasmussen, Emily Shaffer, Josh Steadmon, Adrian Ratiu "git rev-parse --output-object-format=<compat-format>" should print an object compat id when extensions.compatObjectFormat is enabled. Git however doesn't record a compat id for every object, e.g. for objects received by a fetch that goes through index-pack. For such objects, rev-parse silently prints the wrong id: it prints the storage id where the compat id is expected and exits with 0. This happens because rev-parse ignores the repo_oid_to_algop() return value. When no mapping is found, that function fails and leaves the id unchanged, so the storage id is what gets printed. Fix it to die with the same message as pack-objects. The other callers of repo_oid_to_algop() already check its result. This only covers a single revision argument, the one case where rev-parse converts its output. Other forms, such as "A..B" or "A^@", print storage ids whatever --output-object-format says. That is left for a separate change. The test copies a loose object into the repository by hand, so it doesn't depend on whether fetch writes compat mappings. Signed-off-by: Adrian Ratiu <adrian.ratiu@collabora.com> --- builtin/rev-parse.c | 8 +++++--- t/t1500-rev-parse.sh | 18 ++++++++++++++++++ 2 files changed, 23 insertions(+), 3 deletions(-) diff --git a/builtin/rev-parse.c b/builtin/rev-parse.c index d2154eac8d..ab2898ba52 100644 --- a/builtin/rev-parse.c +++ b/builtin/rev-parse.c @@ -1171,9 +1171,11 @@ int cmd_rev_parse(int argc, } if (!repo_get_oid_with_flags(the_repository, name, &oid, flags)) { - if (output_algo) - repo_oid_to_algop(the_repository, &oid, - output_algo, &oid); + if (output_algo && + repo_oid_to_algop(the_repository, &oid, + output_algo, &oid)) + die(_("cannot map object %s to %s"), + oid_to_hex(&oid), output_algo->name); if (verify) revs_count++; else diff --git a/t/t1500-rev-parse.sh b/t/t1500-rev-parse.sh index 30bf8a4d0a..c15d9953a8 100755 --- a/t/t1500-rev-parse.sh +++ b/t/t1500-rev-parse.sh @@ -241,6 +241,24 @@ test_expect_success RUST 'rev-parse --show-object-format in repo with compat mod ) ' +test_expect_success RUST 'rev-parse --output-object-format with unmapped object' ' + test_when_finished "rm -rf plain compat" && + git init --object-format=sha256 plain && + git init --object-format=sha256 compat && + git -C compat config extensions.compatobjectformat sha1 && + oid=$(echo unmapped | git -C plain hash-object -w --stdin) && + # copy the object by hand, so that no compat mapping is written for it + path=$(test_oid_to_path $oid) && + mkdir compat/.git/objects/${path%/*} && + cp plain/.git/objects/$path compat/.git/objects/$path && + echo $oid >expect && + git -C compat rev-parse --output-object-format=sha256 $oid >actual && + test_cmp expect actual && + test_must_fail git -C compat rev-parse --output-object-format=sha1 $oid >actual 2>err && + test_must_be_empty actual && + test_grep "cannot map object $oid to sha1" err +' + test_expect_success 'rev-parse --show-ref-storage-format' ' test_detect_ref_format >expect && git rev-parse --show-ref-storage-format >actual && -- 2.56.1 ^ permalink raw reply related [flat|nested] 10+ messages in thread
* Re: [PATCH 2/3] rev-parse: die when --output-object-format cannot map an object 2026-11-12 19:57 ` [PATCH 2/3] rev-parse: die when --output-object-format cannot map an object Adrian Ratiu @ 2026-11-16 7:37 ` Patrick Steinhardt 2026-11-17 11:55 ` Adrian Ratiu 0 siblings, 1 reply; 10+ messages in thread From: Patrick Steinhardt @ 2026-11-16 7:37 UTC (permalink / raw) To: Adrian Ratiu Cc: git, Junio C Hamano, brian m . carlson, Eric W . Biederman, Axel Rasmussen, Emily Shaffer, Josh Steadmon On Thu, Nov 12, 2026 at 09:57:39PM +0200, Adrian Ratiu wrote: > "git rev-parse --output-object-format=<compat-format>" should print > an object compat id when extensions.compatObjectFormat is enabled. > > Git however doesn't record a compat id for every object, e.g. for > objects received by a fetch that goes through index-pack. > For such objects, rev-parse silently prints the wrong id: it prints > the storage id where the compat id is expected and exits with 0. Yeah, that's just plain wrong and extremely confusing. > This happens because rev-parse ignores the repo_oid_to_algop() return > value. When no mapping is found, that function fails and leaves the > id unchanged, so the storage id is what gets printed. > > Fix it to die with the same message as pack-objects. The other callers > of repo_oid_to_algop() already check its result. > > This only covers a single revision argument, the one case where > rev-parse converts its output. Other forms, such as "A..B" or "A^@", > print storage ids whatever --output-object-format says. That is left > for a separate change. This paragraph is a bit hard to parse. Note that this only covers the case where the user asks for a single revision, as it is the only case where we pass the desired output hash explicitly. All the other cases, like for example when the user passes a range, do not use the desired output hash at all. Fixing that would be a bigger change though, so this is left for a future patch series. > The test copies a loose object into the repository by hand, so it > doesn't depend on whether fetch writes compat mappings. Do we have to do that? I'm mostly asking in the context of the new object storage format extension, which may change where and how exactly objects are stored. So all tests that manually modify the object database will break, and we'll want and need to fix those. > diff --git a/builtin/rev-parse.c b/builtin/rev-parse.c > index d2154eac8d..ab2898ba52 100644 > --- a/builtin/rev-parse.c > +++ b/builtin/rev-parse.c > @@ -1171,9 +1171,11 @@ int cmd_rev_parse(int argc, > } > if (!repo_get_oid_with_flags(the_repository, name, &oid, > flags)) { > - if (output_algo) > - repo_oid_to_algop(the_repository, &oid, > - output_algo, &oid); > + if (output_algo && > + repo_oid_to_algop(the_repository, &oid, > + output_algo, &oid)) > + die(_("cannot map object %s to %s"), > + oid_to_hex(&oid), output_algo->name); Makes sense. I would have checked for `< 0`, but that ultimately doesn't matter much. > diff --git a/t/t1500-rev-parse.sh b/t/t1500-rev-parse.sh > index 30bf8a4d0a..c15d9953a8 100755 > --- a/t/t1500-rev-parse.sh > +++ b/t/t1500-rev-parse.sh > @@ -241,6 +241,24 @@ test_expect_success RUST 'rev-parse --show-object-format in repo with compat mod > ) > ' > > +test_expect_success RUST 'rev-parse --output-object-format with unmapped object' ' > + test_when_finished "rm -rf plain compat" && > + git init --object-format=sha256 plain && > + git init --object-format=sha256 compat && > + git -C compat config extensions.compatobjectformat sha1 && > + oid=$(echo unmapped | git -C plain hash-object -w --stdin) && > + # copy the object by hand, so that no compat mapping is written for it Instead of copying by hand, wouldn't it be possible to write the object with the compat object format unset and then set it afterwards? That should also ensure that we don't have the compat hash. Patrick ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [PATCH 2/3] rev-parse: die when --output-object-format cannot map an object 2026-11-16 7:37 ` Patrick Steinhardt @ 2026-11-17 11:55 ` Adrian Ratiu 0 siblings, 0 replies; 10+ messages in thread From: Adrian Ratiu @ 2026-11-17 11:55 UTC (permalink / raw) To: Patrick Steinhardt Cc: git, Junio C Hamano, brian m . carlson, Eric W . Biederman, Axel Rasmussen, Emily Shaffer, Josh Steadmon On Mon, 16 Nov 2026, Patrick Steinhardt <ps@pks.im> wrote: > On Thu, Nov 12, 2026 at 09:57:39PM +0200, Adrian Ratiu wrote: >> This happens because rev-parse ignores the repo_oid_to_algop() return >> value. When no mapping is found, that function fails and leaves the >> id unchanged, so the storage id is what gets printed. >> >> Fix it to die with the same message as pack-objects. The other callers >> of repo_oid_to_algop() already check its result. >> >> This only covers a single revision argument, the one case where >> rev-parse converts its output. Other forms, such as "A..B" or "A^@", >> print storage ids whatever --output-object-format says. That is left >> for a separate change. > > This paragraph is a bit hard to parse. > > Note that this only covers the case where the user asks for a single > revision, as it is the only case where we pass the desired output hash > explicitly. All the other cases, like for example when the user passes > a range, do not use the desired output hash at all. Fixing that would > be a bigger change though, so this is left for a future patch series. Ack, I'll reword it. >> The test copies a loose object into the repository by hand, so it >> doesn't depend on whether fetch writes compat mappings. > > Do we have to do that? I'm mostly asking in the context of the new > object storage format extension, which may change where and how exactly > objects are stored. So all tests that manually modify the object > database will break, and we'll want and need to fix those. We can do it any other way, no problem. How I reproduced this in the real world (non-test) is by creating a commit with JJ, but I tried to avoid adding JJ as a test dependency. :) Besides, I'll fix this in newer JJ as well, to simply reject repos with extensions.compatObjectFormat, which is not supported either in Gitoxide or JJ. >> diff --git a/t/t1500-rev-parse.sh b/t/t1500-rev-parse.sh >> index 30bf8a4d0a..c15d9953a8 100755 >> --- a/t/t1500-rev-parse.sh >> +++ b/t/t1500-rev-parse.sh >> @@ -241,6 +241,24 @@ test_expect_success RUST 'rev-parse --show-object-format in repo with compat mod >> ) >> ' >> >> +test_expect_success RUST 'rev-parse --output-object-format with unmapped object' ' >> + test_when_finished "rm -rf plain compat" && >> + git init --object-format=sha256 plain && >> + git init --object-format=sha256 compat && >> + git -C compat config extensions.compatobjectformat sha1 && >> + oid=$(echo unmapped | git -C plain hash-object -w --stdin) && >> + # copy the object by hand, so that no compat mapping is written for it > > Instead of copying by hand, wouldn't it be possible to write the object > with the compat object format unset and then set it afterwards? That > should also ensure that we don't have the compat hash. Yes, I just verified that it works. Will it this way in v2, thanks! ^ permalink raw reply [flat|nested] 10+ messages in thread
* [PATCH 3/3] object-file-convert: copy unknown commit headers unchanged 2026-11-12 19:57 [PATCH 0/3] compatObjectFormat: fix a BUG(), wrong id and unknown headers Adrian Ratiu 2026-11-12 19:57 ` [PATCH 1/3] setup: reject compatObjectFormat == objectFormat Adrian Ratiu 2026-11-12 19:57 ` [PATCH 2/3] rev-parse: die when --output-object-format cannot map an object Adrian Ratiu @ 2026-11-12 19:57 ` Adrian Ratiu 2026-11-16 7:37 ` Patrick Steinhardt 2 siblings, 1 reply; 10+ messages in thread From: Adrian Ratiu @ 2026-11-12 19:57 UTC (permalink / raw) To: git Cc: Junio C Hamano, Patrick Steinhardt, brian m . carlson, Eric W . Biederman, Axel Rasmussen, Emily Shaffer, Josh Steadmon, Adrian Ratiu In a repo with extensions.compatObjectFormat, Git converts commits to the compat format in two places which disagree about headers they don't know: 1. When a commit is created with commit_tree_extended(), as "git commit" does, convert_commit_extra_headers() copies unknown headers unchanged. 2. When an existing commit object is converted (git hash-object -w) or when it's read by its compat id, convert_commit_object() fails on them instead. So git hash-object refuses such a commit, while git commit --amend records a compat id for it that can't be read back: "Failed to convert object from sha256 to sha1". The rejection was added in 08a45903cb (object-file-convert: convert commits that embed signed tags, 2023-10-01). Git can't know what an unknown header holds. It might hold an object id that needs converting, so convert_commit_object() rejects all of them. Some tools rely on such headers though. Jujutsu, for example, writes a "change-id" header in every commit, so none of its commits can be converted. Make convert_commit_object() copy unknown headers and their continuation lines unchanged, just like convert_commit_extra_headers() so both give the same compat object. The author, committer, encoding and gpgsig headers were already copied the same, so their separate cases are no longer needed. A "tree", "parent" or "mergetag" line with nothing after the keyword used to be rejected as unknown and is now copied as well. Such a commit is malformed in both formats anyway. Alternatives considered: - Reject unknown headers in convert_commit_extra_headers() too. Both paths would then agree, but commits from tools that add their own headers could not be used in a compat repo. - Copy only headers known to hold no object ids. Every tool would first need to get its headers added to that list. The test writes a commit with an unknown header with "git hash-object", amends it with "git commit", and checks that both commits map to the same objects in the SHA-1 and the SHA-256 repo. Signed-off-by: Adrian Ratiu <adrian.ratiu@collabora.com> --- object-file-convert.c | 21 +++++++++++---------- t/t1016-compatObjectFormat.sh | 18 ++++++++++++++++++ 2 files changed, 29 insertions(+), 10 deletions(-) diff --git a/object-file-convert.c b/object-file-convert.c index 63ee18630b..1a0f98c9ff 100644 --- a/object-file-convert.c +++ b/object-file-convert.c @@ -223,17 +223,18 @@ static int convert_commit_object(struct repository *repo, strbuf_release(&tag); strbuf_release(&new_tag); } - else if (((bufptr + 7) < tail) && !memcmp(bufptr, "author ", 7)) - strbuf_add(out, bufptr, (eol - bufptr) + 1); - else if (((bufptr + 10) < tail) && !memcmp(bufptr, "committer ", 10)) - strbuf_add(out, bufptr, (eol - bufptr) + 1); - else if (((bufptr + 9) < tail) && !memcmp(bufptr, "encoding ", 9)) - strbuf_add(out, bufptr, (eol - bufptr) + 1); - else if (((bufptr + 6) < tail) && !memcmp(bufptr, "gpgsig", 6)) - strbuf_add(out, bufptr, (eol - bufptr) + 1); else { - /* Unknown line fail it might embed an oid */ - return -1; + /* + * Copy all other headers unchanged. author, committer, encoding and gpgsig + * hold no object ids. For unknown headers we can't tell, so copy them too, + * like convert_commit_extra_headers() does when the commit is written. Both + * must give the same compat object. + * + * This means that a header holding object ids is copied without converting + * them. Any such header added to Git needs its own case above, like parent + * or mergetag, and the same conversion in convert_commit_extra_headers(). + */ + strbuf_add(out, bufptr, (eol - bufptr) + 1); } /* Consume any trailing continuation lines */ bufptr = eol + 1; diff --git a/t/t1016-compatObjectFormat.sh b/t/t1016-compatObjectFormat.sh index 9cafcee509..fde092a017 100755 --- a/t/t1016-compatObjectFormat.sh +++ b/t/t1016-compatObjectFormat.sh @@ -187,6 +187,22 @@ do eval signedtag3_${hash}_oid=$(git hash-object -t tag -w ../${hash}_signedtag3) && eval signedtag4_${hash}_oid=$(git hash-object -t tag -w ../${hash}_signedtag4) ' + test_expect_success "create a $hash commit with an unknown header" ' + git cat-file commit $(commit2_oid $hash) >../${hash}_commit2 && + awk "{ print } /^committer / { print \"x-unknown one\"; print \" two\" }" \ + <../${hash}_commit2 >../${hash}_unknownheader && + oid=$(git hash-object -t commit -w ../${hash}_unknownheader) && + eval unknownheader_${hash}_oid=$oid && + git branch unknownheader $oid + ' + test_expect_success "amend a $hash commit with an unknown header" ' + git checkout unknownheader && + git commit --amend -m "Amend a commit with an unknown header" && + git cat-file commit HEAD >actual && + test_grep "^x-unknown one$" actual && + oid=$(git rev-parse HEAD) && + eval unknownheader2_${hash}_oid=$oid + ' test_expect_success 'rev-parse maps oid of object borrowed from alternate' ' for repo in alt borrow @@ -298,6 +314,8 @@ compare_oids 'blob' more "$more_sha1_oid" "$more_sha256_oid" compare_oids 'blob' another "$another_sha1_oid" "$another_sha256_oid" compare_oids 'tree' tree2 "$tree2_sha1_oid" "$tree2_sha256_oid" compare_oids 'commit' commit2 "$commit2_sha1_oid" "$commit2_sha256_oid" +compare_oids 'commit' unknownheader "$unknownheader_sha1_oid" "$unknownheader_sha256_oid" +compare_oids 'commit' unknownheader2 "$unknownheader2_sha1_oid" "$unknownheader2_sha256_oid" compare_oids GPG2 'tag' signedtag2 "$signedtag2_sha1_oid" "$signedtag2_sha256_oid" compare_oids GPG2 'commit' signedcommit2 "$signedcommit2_sha1_oid" "$signedcommit2_sha256_oid" compare_oids GPG2 'commit' signedcommit3 "$signedcommit3_sha1_oid" "$signedcommit3_sha256_oid" -- 2.56.1 ^ permalink raw reply related [flat|nested] 10+ messages in thread
* Re: [PATCH 3/3] object-file-convert: copy unknown commit headers unchanged 2026-11-12 19:57 ` [PATCH 3/3] object-file-convert: copy unknown commit headers unchanged Adrian Ratiu @ 2026-11-16 7:37 ` Patrick Steinhardt 2026-11-17 15:27 ` Adrian Ratiu 0 siblings, 1 reply; 10+ messages in thread From: Patrick Steinhardt @ 2026-11-16 7:37 UTC (permalink / raw) To: Adrian Ratiu Cc: git, Junio C Hamano, brian m . carlson, Eric W . Biederman, Axel Rasmussen, Emily Shaffer, Josh Steadmon On Thu, Nov 12, 2026 at 09:57:40PM +0200, Adrian Ratiu wrote: > In a repo with extensions.compatObjectFormat, Git converts commits > to the compat format in two places which disagree about headers they > don't know: > > 1. When a commit is created with commit_tree_extended(), as > "git commit" does, convert_commit_extra_headers() copies unknown > headers unchanged. > > 2. When an existing commit object is converted (git hash-object -w) > or when it's read by its compat id, convert_commit_object() fails > on them instead. > > So git hash-object refuses such a commit, while git commit --amend s/git hash-object/git-hash-object(1)/ > records a compat id for it that can't be read back: s/compat id/compat ID/ > "Failed to convert object from sha256 to sha1". > > The rejection was added in 08a45903cb (object-file-convert: convert > commits that embed signed tags, 2023-10-01). Git can't know what an > unknown header holds. It might hold an object id that needs s/object id/object ID/ > converting, so convert_commit_object() rejects all of them. > > Some tools rely on such headers though. Jujutsu, for example, writes > a "change-id" header in every commit, so none of its commits can be > converted. > > Make convert_commit_object() copy unknown headers and their > continuation lines unchanged, just like convert_commit_extra_headers() > so both give the same compat object. > > The author, committer, encoding and gpgsig headers were already copied > the same, so their separate cases are no longer needed. > > A "tree", "parent" or "mergetag" line with nothing after the keyword > used to be rejected as unknown and is now copied as well. Such a > commit is malformed in both formats anyway. Hmm. I guess this here might raise some eyebrows. I guess it makes the code simpler, but I wonder what the benefit is of converting an object that is so obviously corrupt... ... yeah, after reading the code we'd have to introduce a bunch of extra logic to make this work as expected. As far as I can see though we _do_ reject "tree " without trailing data, but we wouldn't reject "tree" without the trailing space. We could fix that by introducing a generic format check: - The line must have at least a single space separating key and value of the header. - The key must only contain characters of a specific set. Otherwise we might be too lenient and also accept data that doesn't even look close to an object header. But whether that's all worth it is a different question, as said. It's a corrupt commit anyway, and we cannot address the root cause of that corruption here anyway. > Alternatives considered: > > - Reject unknown headers in convert_commit_extra_headers() too. > Both paths would then agree, but commits from tools that add > their own headers could not be used in a compat repo. Yeah, that's certainly a direction we shouldn't go. There's way too many tools nowadays with widespread usage that write such unknown headers. > - Copy only headers known to hold no object ids. Every tool would > first need to get its headers added to that list. Agreed, that doesn't seem sensible, either. We don't want to be the gatekeeper of who is or who isn't allowed to have their own custom header. A third alternative could have been to try and recognize commit IDs automatically. E.g. if you see a 40-character hex string then we treat it as a commit ID. But I don't think that's sensible, either, as we shouldn't interpret what we don't know. A bit of a side note, but I have been wondering multiple times already whether we should fix this gap by introducing a well-known object header prefix that tells us that the value of that header is indeed an object ID, for example something like "x-obj-foobar <oid>". That would make our commit graph extensible by third-party-tools without us having to understand what that tool even intends to do with that, and we would know to treat such object IDs as reachable. But that's of course a topic that is well outside the scope of this patch series. > diff --git a/object-file-convert.c b/object-file-convert.c > index 63ee18630b..1a0f98c9ff 100644 > --- a/object-file-convert.c > +++ b/object-file-convert.c > @@ -223,17 +223,18 @@ static int convert_commit_object(struct repository *repo, > strbuf_release(&tag); > strbuf_release(&new_tag); > } > - else if (((bufptr + 7) < tail) && !memcmp(bufptr, "author ", 7)) > - strbuf_add(out, bufptr, (eol - bufptr) + 1); > - else if (((bufptr + 10) < tail) && !memcmp(bufptr, "committer ", 10)) > - strbuf_add(out, bufptr, (eol - bufptr) + 1); > - else if (((bufptr + 9) < tail) && !memcmp(bufptr, "encoding ", 9)) > - strbuf_add(out, bufptr, (eol - bufptr) + 1); > - else if (((bufptr + 6) < tail) && !memcmp(bufptr, "gpgsig", 6)) > - strbuf_add(out, bufptr, (eol - bufptr) + 1); > else { > - /* Unknown line fail it might embed an oid */ > - return -1; > + /* > + * Copy all other headers unchanged. author, committer, encoding and gpgsig > + * hold no object ids. For unknown headers we can't tell, so copy them too, s/object ids/object IDs/ > + * like convert_commit_extra_headers() does when the commit is written. Both > + * must give the same compat object. > + * > + * This means that a header holding object ids is copied without converting Likewise. > + * them. Any such header added to Git needs its own case above, like parent > + * or mergetag, and the same conversion in convert_commit_extra_headers(). > + */ > + strbuf_add(out, bufptr, (eol - bufptr) + 1); > } > /* Consume any trailing continuation lines */ > bufptr = eol + 1; > diff --git a/t/t1016-compatObjectFormat.sh b/t/t1016-compatObjectFormat.sh > index 9cafcee509..fde092a017 100755 > --- a/t/t1016-compatObjectFormat.sh > +++ b/t/t1016-compatObjectFormat.sh > @@ -187,6 +187,22 @@ do > eval signedtag3_${hash}_oid=$(git hash-object -t tag -w ../${hash}_signedtag3) && > eval signedtag4_${hash}_oid=$(git hash-object -t tag -w ../${hash}_signedtag4) > ' Let's have an empty line between tests. > + test_expect_success "create a $hash commit with an unknown header" ' > + git cat-file commit $(commit2_oid $hash) >../${hash}_commit2 && > + awk "{ print } /^committer / { print \"x-unknown one\"; print \" two\" }" \ > + <../${hash}_commit2 >../${hash}_unknownheader && > + oid=$(git hash-object -t commit -w ../${hash}_unknownheader) && > + eval unknownheader_${hash}_oid=$oid && > + git branch unknownheader $oid > + ' > + test_expect_success "amend a $hash commit with an unknown header" ' > + git checkout unknownheader && > + git commit --amend -m "Amend a commit with an unknown header" && > + git cat-file commit HEAD >actual && > + test_grep "^x-unknown one$" actual && > + oid=$(git rev-parse HEAD) && > + eval unknownheader2_${hash}_oid=$oid > + ' > > test_expect_success 'rev-parse maps oid of object borrowed from alternate' ' > for repo in alt borrow Thanks! Patrick ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [PATCH 3/3] object-file-convert: copy unknown commit headers unchanged 2026-11-16 7:37 ` Patrick Steinhardt @ 2026-11-17 15:27 ` Adrian Ratiu 0 siblings, 0 replies; 10+ messages in thread From: Adrian Ratiu @ 2026-11-17 15:27 UTC (permalink / raw) To: Patrick Steinhardt Cc: git, Junio C Hamano, brian m . carlson, Eric W . Biederman, Axel Rasmussen, Emily Shaffer, Josh Steadmon On Mon, 16 Nov 2026, Patrick Steinhardt <ps@pks.im> wrote: > On Thu, Nov 12, 2026 at 09:57:40PM +0200, Adrian Ratiu wrote: >> In a repo with extensions.compatObjectFormat, Git converts commits >> to the compat format in two places which disagree about headers they >> don't know: >> >> 1. When a commit is created with commit_tree_extended(), as >> "git commit" does, convert_commit_extra_headers() copies unknown >> headers unchanged. >> >> 2. When an existing commit object is converted (git hash-object -w) >> or when it's read by its compat id, convert_commit_object() fails >> on them instead. >> >> So git hash-object refuses such a commit, while git commit --amend > > s/git hash-object/git-hash-object(1)/ > >> records a compat id for it that can't be read back: > > s/compat id/compat ID/ > >> "Failed to convert object from sha256 to sha1". >> >> The rejection was added in 08a45903cb (object-file-convert: convert >> commits that embed signed tags, 2023-10-01). Git can't know what an >> unknown header holds. It might hold an object id that needs > > s/object id/object ID/ > Ack on these and all the other ID / spacing fixes, will do them in v2. >> converting, so convert_commit_object() rejects all of them. >> >> Some tools rely on such headers though. Jujutsu, for example, writes >> a "change-id" header in every commit, so none of its commits can be >> converted. >> >> Make convert_commit_object() copy unknown headers and their >> continuation lines unchanged, just like convert_commit_extra_headers() >> so both give the same compat object. >> >> The author, committer, encoding and gpgsig headers were already copied >> the same, so their separate cases are no longer needed. >> >> A "tree", "parent" or "mergetag" line with nothing after the keyword >> used to be rejected as unknown and is now copied as well. Such a >> commit is malformed in both formats anyway. > > Hmm. I guess this here might raise some eyebrows. I guess it makes the > code simpler, but I wonder what the benefit is of converting an object > that is so obviously corrupt... > > ... yeah, after reading the code we'd have to introduce a bunch of extra > logic to make this work as expected. As far as I can see though we _do_ > reject "tree " without trailing data, but we wouldn't reject "tree" > without the trailing space. I looked closer and v1 actually copies "tree " too, because of the (bufptr + 5) < eol check, "tree \n" never reaches the tree branch. :) I also found I made mergetag worse: a "mergetag " line with an empty first line and the tag in continuation lines is also copied, so its embedded object ID ends up unconverted in the compat object. Git still treats that header as a mergetag ("git show" reports "malformed mergetag") and "git commit --amend" refuses to convert it, so v1 makes the two paths disagree again, just in the other direction. I can fix this in v2 by changing the three checks to <=, so the converter classifies headers same as read_commit_extra_header_lines(), by the key before the first space. So: "tree ", "parent " and "mergetag " with an empty or bad value are rejected, as before this series. "tree", "parent" and "mergetag" without a space are not those headers as far as Git is concerned, so they are copied like any unknown header. That is also what --amend does with them. I'll also add a test for the empty mergetag case. > > We could fix that by introducing a generic format check: > > - The line must have at least a single space separating key and value > of the header. > > - The key must only contain characters of a specific set. > > Otherwise we might be too lenient and also accept data that doesn't even > look close to an object header. > > But whether that's all worth it is a different question, as said. It's a > corrupt commit anyway, and we cannot address the root cause of that > corruption here anyway. With the above I don't think we need it. A check only in the converter would reject headers that read_commit_extra_header_lines() accepts and "git commit --amend" keeps (e.g. bare "x-foo" line), so amend would again record compat IDs that can't be read back. If we need such rules, I think fsck is the better place (today it doesn't look at any header after "committer"). It's a separate topic though. >> Alternatives considered: >> >> - Reject unknown headers in convert_commit_extra_headers() too. >> Both paths would then agree, but commits from tools that add >> their own headers could not be used in a compat repo. > > Yeah, that's certainly a direction we shouldn't go. There's way too many > tools nowadays with widespread usage that write such unknown headers. > >> - Copy only headers known to hold no object ids. Every tool would >> first need to get its headers added to that list. > > Agreed, that doesn't seem sensible, either. We don't want to be the > gatekeeper of who is or who isn't allowed to have their own custom > header. > > A third alternative could have been to try and recognize commit IDs > automatically. E.g. if you see a 40-character hex string then we treat > it as a commit ID. But I don't think that's sensible, either, as we > shouldn't interpret what we don't know. Agreed, unknown unknowns are such a pain. > > A bit of a side note, but I have been wondering multiple times already > whether we should fix this gap by introducing a well-known object header > prefix that tells us that the value of that header is indeed an object > ID, for example something like "x-obj-foobar <oid>". That would make our > commit graph extensible by third-party-tools without us having to > understand what that tool even intends to do with that, and we would > know to treat such object IDs as reachable. > > But that's of course a topic that is well outside the scope of this > patch series. Yes, I'm trying to keep this series small and just address the immediate bugs / breakages, to avoid it balooning into 10+ patch series again. :) ^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2026-11-17 15:28 UTC | newest] Thread overview: 10+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2026-11-12 19:57 [PATCH 0/3] compatObjectFormat: fix a BUG(), wrong id and unknown headers Adrian Ratiu 2026-11-12 19:57 ` [PATCH 1/3] setup: reject compatObjectFormat == objectFormat Adrian Ratiu 2026-11-16 7:36 ` Patrick Steinhardt 2026-11-17 10:42 ` Adrian Ratiu 2026-11-12 19:57 ` [PATCH 2/3] rev-parse: die when --output-object-format cannot map an object Adrian Ratiu 2026-11-16 7:37 ` Patrick Steinhardt 2026-11-17 11:55 ` Adrian Ratiu 2026-11-12 19:57 ` [PATCH 3/3] object-file-convert: copy unknown commit headers unchanged Adrian Ratiu 2026-11-16 7:37 ` Patrick Steinhardt 2026-11-17 15:27 ` Adrian Ratiu
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.