Git development
 help / color / mirror / Atom feed
From: Adrian Ratiu <adrian.ratiu@collabora.com>
To: git@vger.kernel.org
Cc: Junio C Hamano <gitster@pobox.com>,
	Patrick Steinhardt <ps@pks.im>,
	"Eric W . Biederman" <ebiederm@xmission.com>,
	"brian m . carlson" <sandals@crustytoothpaste.net>,
	Adrian Ratiu <adrian.ratiu@collabora.com>
Subject: [PATCH v2 0/3] compatObjectFormat: fix a BUG(), wrong ID and unknown headers
Date: Thu, 19 Nov 2026 16:18:19 +0200	[thread overview]
Message-ID: <cover.1794932695.git.adrian.ratiu@collabora.com> (raw)
In-Reply-To: <20261112195740.2834341-1-adrian.ratiu@collabora.com>

Hello all,

This is v2 of the compatObjectFormat fixes found while testing sha1-sha256
interop between Git, Gitoxide and Jujutsu.

Because patch 3 drops a check that 08a45903cb (object-file-convert: convert
commits that embed signed tags, 2023-10-01) added on purpose, I'd still welcome
a look from Eric or Brian.

Changes in v2:
 * Check repo_oid_to_algop() for "< 0" and reword the note on which
   rev-parse forms are covered. The test now writes the object before
   enabling compat instead of copying it into the object directory by
   hand. (Patrick)
 * Match the "tree ", "parent " and "mergetag " headers by their key,
   like read_commit_extra_header_lines() does. (Patrick)
 * Minor casing, spacing, doc and commit msg fixes (Patrick)

v1: https://lore.kernel.org/git/20261112195740.2834341-1-adrian.ratiu@collabora.com/

Many thanks,
Adrian

Adrian Ratiu (3):
  setup: reject compatObjectFormat == objectFormat
  rev-parse: die when --output-object-format cannot map an object
  object-file-convert: copy unknown commit headers unchanged

 builtin/rev-parse.c           |  8 +++++---
 object-file-convert.c         | 35 ++++++++++++++++++++++-------------
 setup.c                       |  7 +++++++
 t/t1016-compatObjectFormat.sh | 34 ++++++++++++++++++++++++++++++++++
 t/t1302-repo-version.sh       | 14 ++++++++++++++
 t/t1500-rev-parse.sh          | 14 ++++++++++++++
 6 files changed, 96 insertions(+), 16 deletions(-)

Range-diff against v1:
1:  53c814a1e1 ! 1:  f74a0fd67a setup: reject compatObjectFormat == objectFormat
    @@ setup.c: int verify_repository_format(const struct repository_format *format,
      
     +	if (format->compat_hash_algo == format->hash_algo) {
     +		strbuf_addf(err, _("'%s' is '%s', the same as the object format"),
    -+			    "extensions.compatobjectformat",
    ++			    "extensions.compatObjectFormat",
     +			    hash_algos[format->hash_algo].name);
     +		return -1;
     +	}
    @@ t/t1302-repo-version.sh: abort 0 noop-v1
     +	test_expect_success "compatObjectFormat equal to objectFormat $hash" "
     +		test_when_finished 'rm -rf same' &&
     +		git init --object-format=$hash same &&
    -+		git -C same config core.repositoryformatversion 1 &&
    -+		git -C same config extensions.compatObjectFormat $hash &&
    ++		git -C same config set core.repositoryformatversion 1 &&
    ++		git -C same config set extensions.compatObjectFormat $hash &&
     +		test_must_fail git -C same rev-parse --git-dir 2>err &&
    -+		test_grep \"'extensions.compatobjectformat' is '$hash', the same as the object format\" err &&
    -+		git config --file=same/.git/config --unset extensions.compatObjectFormat &&
    ++		test_grep \"'extensions.compatObjectFormat' is '$hash', the same as the object format\" err &&
    ++		git config unset --file=same/.git/config extensions.compatObjectFormat &&
     +		git -C same rev-parse --git-dir
     +	"
     +done
2:  066fc59207 ! 2:  5b91372748 rev-parse: die when --output-object-format cannot map an object
    @@ Commit message
         rev-parse: die when --output-object-format cannot map an object
     
         "git rev-parse --output-object-format=<compat-format>" should print
    -    an object compat id when extensions.compatObjectFormat is enabled.
    +    the compat ID of an object when extensions.compatObjectFormat is
    +    enabled.
     
    -    Git however doesn't record a compat id for every object, e.g. for
    +    Git however doesn't record a compat ID for every object, e.g. for
         objects received by a fetch that goes through index-pack.
    -    For such objects, rev-parse silently prints the wrong id: it prints
    -    the storage id where the compat id is expected and exits with 0.
    +    For such objects, rev-parse silently prints the wrong ID: it prints
    +    the storage ID where the compat ID is expected and exits with 0.
     
         This happens because rev-parse ignores the repo_oid_to_algop() return
         value. When no mapping is found, that function fails and leaves the
    -    id unchanged, so the storage id is what gets printed.
    +    ID unchanged, so the storage ID is what gets printed.
     
         Fix it to die with the same message as pack-objects. The other callers
         of repo_oid_to_algop() already check its result.
     
    -    This only covers a single revision argument, the one case where
    -    rev-parse converts its output. Other forms, such as "A..B" or "A^@",
    -    print storage ids whatever --output-object-format says. That is left
    -    for a separate change.
    -
    -    The test copies a loose object into the repository by hand, so it
    -    doesn't depend on whether fetch writes compat mappings.
    +    Note that this only covers the case where the user asks for a single
    +    revision, as it is the only case where rev-parse passes the desired
    +    output hash explicitly. All the other cases, like for example when the
    +    user passes a range, do not use the desired output hash at all. Fixing
    +    that would be a bigger change though, so it is left for a future patch
    +    series.
     
         Signed-off-by: Adrian Ratiu <adrian.ratiu@collabora.com>
     
    @@ builtin/rev-parse.c: int cmd_rev_parse(int argc,
     -						  output_algo, &oid);
     +			if (output_algo &&
     +			    repo_oid_to_algop(the_repository, &oid,
    -+					      output_algo, &oid))
    ++					      output_algo, &oid) < 0)
     +				die(_("cannot map object %s to %s"),
     +				    oid_to_hex(&oid), output_algo->name);
      			if (verify)
    @@ t/t1500-rev-parse.sh: test_expect_success RUST 'rev-parse --show-object-format i
      '
      
     +test_expect_success RUST 'rev-parse --output-object-format with unmapped object' '
    -+	test_when_finished "rm -rf plain compat" &&
    -+	git init --object-format=sha256 plain &&
    -+	git init --object-format=sha256 compat &&
    -+	git -C compat config extensions.compatobjectformat sha1 &&
    -+	oid=$(echo unmapped | git -C plain hash-object -w --stdin) &&
    -+	# copy the object by hand, so that no compat mapping is written for it
    -+	path=$(test_oid_to_path $oid) &&
    -+	mkdir compat/.git/objects/${path%/*} &&
    -+	cp plain/.git/objects/$path compat/.git/objects/$path &&
    ++	test_when_finished "rm -rf unmapped" &&
    ++	git init --object-format=sha256 unmapped &&
    ++	# write the object before enabling compat, so it gets no compat mapping
    ++	oid=$(echo unmapped | git -C unmapped hash-object -w --stdin) &&
    ++	git -C unmapped config set extensions.compatObjectFormat sha1 &&
     +	echo $oid >expect &&
    -+	git -C compat rev-parse --output-object-format=sha256 $oid >actual &&
    ++	git -C unmapped rev-parse --output-object-format=sha256 $oid >actual &&
     +	test_cmp expect actual &&
    -+	test_must_fail git -C compat rev-parse --output-object-format=sha1 $oid >actual 2>err &&
    ++	test_must_fail git -C unmapped rev-parse --output-object-format=sha1 $oid >actual 2>err &&
     +	test_must_be_empty actual &&
     +	test_grep "cannot map object $oid to sha1" err
     +'
3:  a171e86b54 ! 3:  26080a7a38 object-file-convert: copy unknown commit headers unchanged
    @@ Commit message
         don't know:
     
         1. When a commit is created with commit_tree_extended(), as
    -       "git commit" does, convert_commit_extra_headers() copies unknown
    +       git-commit(1) does, convert_commit_extra_headers() copies unknown
            headers unchanged.
     
    -    2. When an existing commit object is converted (git hash-object -w)
    -       or when it's read by its compat id, convert_commit_object() fails
    +    2. When an existing commit object is converted (git-hash-object(1) -w)
    +       or when it's read by its compat ID, convert_commit_object() fails
            on them instead.
     
    -    So git hash-object refuses such a commit, while git commit --amend
    -    records a compat id for it that can't be read back:
    +    So git-hash-object(1) refuses such a commit, while "git commit --amend"
    +    records a compat ID for it that can't be read back:
            "Failed to convert object from sha256 to sha1".
     
         The rejection was added in 08a45903cb (object-file-convert: convert
         commits that embed signed tags, 2023-10-01). Git can't know what an
    -    unknown header holds. It might hold an object id that needs
    +    unknown header holds. It might hold an object ID that needs
         converting, so convert_commit_object() rejects all of them.
     
         Some tools rely on such headers though. Jujutsu, for example, writes
    @@ Commit message
         converted.
     
         Make convert_commit_object() copy unknown headers and their
    -    continuation lines unchanged, just like convert_commit_extra_headers()
    +    continuation lines unchanged, just like convert_commit_extra_headers(),
         so both give the same compat object.
     
         The author, committer, encoding and gpgsig headers were already copied
         the same, so their separate cases are no longer needed.
     
    -    A "tree", "parent" or "mergetag" line with nothing after the keyword
    -    used to be rejected as unknown and is now copied as well. Such a
    -    commit is malformed in both formats anyway.
    +    The "tree ", "parent " and "mergetag " cases only matched a line with
    +    something after the space. That didn't matter while all other lines
    +    were rejected, but they would now be copied. For a "mergetag " line
    +    whose tag starts on the next line, the object ID in the tag would be
    +    copied unconverted, even though Git reads that header as a mergetag and
    +    convert_commit_extra_headers() refuses to convert it.
    +
    +    Match these headers by their key alone instead, the way
    +    read_commit_extra_header_lines() does. Such a line with an empty or bad
    +    value is then rejected, as before. A "tree", "parent" or "mergetag"
    +    line without a space is not one of these headers to Git, so it is
    +    copied like any other unknown header, which is also what
    +    convert_commit_extra_headers() does.
     
         Alternatives considered:
     
    @@ Commit message
            Both paths would then agree, but commits from tools that add
            their own headers could not be used in a compat repo.
     
    -     - Copy only headers known to hold no object ids. Every tool would
    +     - Copy only headers known to hold no object IDs. Every tool would
            first need to get its headers added to that list.
     
    -    The test writes a commit with an unknown header with "git hash-object",
    -    amends it with "git commit", and checks that both commits map to the
    -    same objects in the SHA-1 and the SHA-256 repo.
    +     - Convert values that look like object IDs. Any other hex string
    +       could be taken for one, and Git shouldn't interpret headers it
    +       doesn't know.
    +
    +     - Also reject lines that don't look like "<key> <value>". This would
    +       make the two paths disagree again, because
    +       convert_commit_extra_headers() copies such lines, so
    +       "git commit --amend" would record compat IDs that can't be read
    +       back.
     
         Signed-off-by: Adrian Ratiu <adrian.ratiu@collabora.com>
     
      ## object-file-convert.c ##
    +@@ object-file-convert.c: static int convert_commit_object(struct repository *repo,
    + 		if (!eol)
    + 			return error(_("bad %s in commit"), "line");
    + 
    +-		if (((bufptr + 5) < eol) && !memcmp(bufptr, "tree ", 5))
    ++		if (((bufptr + 5) <= eol) && !memcmp(bufptr, "tree ", 5))
    + 		{
    + 			if (((bufptr + tree_entry_len) != eol) ||
    + 			    parse_oid_hex_algop(bufptr + 5, &oid, &p, from) ||
    +@@ object-file-convert.c: static int convert_commit_object(struct repository *repo,
    + 					     "tree", oid_to_hex(&oid));
    + 			strbuf_addf(out, "tree %s\n", oid_to_hex(&mapped_oid));
    + 		}
    +-		else if (((bufptr + 7) < eol) && !memcmp(bufptr, "parent ", 7))
    ++		else if (((bufptr + 7) <= eol) && !memcmp(bufptr, "parent ", 7))
    + 		{
    + 			if (((bufptr + parent_entry_len) != eol) ||
    + 			    parse_oid_hex_algop(bufptr + 7, &oid, &p, from) ||
    +@@ object-file-convert.c: static int convert_commit_object(struct repository *repo,
    + 
    + 			strbuf_addf(out, "parent %s\n", oid_to_hex(&mapped_oid));
    + 		}
    +-		else if (((bufptr + 9) < eol) && !memcmp(bufptr, "mergetag ", 9))
    ++		else if (((bufptr + 9) <= eol) && !memcmp(bufptr, "mergetag ", 9))
    + 		{
    + 			struct strbuf tag = STRBUF_INIT, new_tag = STRBUF_INIT;
    + 
     @@ object-file-convert.c: static int convert_commit_object(struct repository *repo,
      			strbuf_release(&tag);
      			strbuf_release(&new_tag);
    @@ object-file-convert.c: static int convert_commit_object(struct repository *repo,
     -			/* Unknown line fail it might embed an oid */
     -			return -1;
     +			/*
    -+			 * Copy all other headers unchanged. author, committer, encoding and gpgsig
    -+			 * hold no object ids. For unknown headers we can't tell, so copy them too,
    -+			 * like convert_commit_extra_headers() does when the commit is written. Both
    -+			 * must give the same compat object.
    ++			 * Copy all other headers unchanged. author, committer,
    ++			 * encoding and gpgsig hold no object IDs. For unknown
    ++			 * headers we can't tell, so copy them too, like
    ++			 * convert_commit_extra_headers() does when the commit
    ++			 * is written. Both must give the same compat object.
    ++			 *
    ++			 * This means that a header holding object IDs is
    ++			 * copied without converting them. Any such header added
    ++			 * to Git needs its own case above, like parent or
    ++			 * mergetag, and the same conversion in
    ++			 * convert_commit_extra_headers().
     +			 *
    -+			 * This means that a header holding object ids is copied without converting
    -+			 * them. Any such header added to Git needs its own case above, like parent
    -+			 * or mergetag, and the same conversion in convert_commit_extra_headers().
    ++			 * The cases above match a header by its key alone, as
    ++			 * read_commit_extra_header_lines() does, so that e.g. a
    ++			 * "mergetag " line with nothing after the space is
    ++			 * still handled as a mergetag instead of being copied.
     +			 */
     +			strbuf_add(out, bufptr, (eol - bufptr) + 1);
      		}
    @@ object-file-convert.c: static int convert_commit_object(struct repository *repo,
     
      ## t/t1016-compatObjectFormat.sh ##
     @@ t/t1016-compatObjectFormat.sh: do
    - 		eval signedtag3_${hash}_oid=$(git hash-object -t tag -w ../${hash}_signedtag3) &&
      		eval signedtag4_${hash}_oid=$(git hash-object -t tag -w ../${hash}_signedtag4)
      	'
    + 
     +	test_expect_success "create a $hash commit with an unknown header" '
     +		git cat-file commit $(commit2_oid $hash) >../${hash}_commit2 &&
     +		awk "{ print } /^committer / { print \"x-unknown one\"; print \" two\" }" \
    @@ t/t1016-compatObjectFormat.sh: do
     +		eval unknownheader_${hash}_oid=$oid &&
     +		git branch unknownheader $oid
     +	'
    ++
     +	test_expect_success "amend a $hash commit with an unknown header" '
     +		git checkout unknownheader &&
     +		git commit --amend -m "Amend a commit with an unknown header" &&
    @@ t/t1016-compatObjectFormat.sh: do
     +		oid=$(git rev-parse HEAD) &&
     +		eval unknownheader2_${hash}_oid=$oid
     +	'
    - 
    ++
    ++	test_expect_success "reject a $hash commit with an empty mergetag line" '
    ++		git cat-file tag committag >../${hash}_committag &&
    ++		{
    ++			sed -n "/^committer /q;p" ../${hash}_commit2 &&
    ++			grep "^committer " ../${hash}_commit2 &&
    ++			echo "mergetag " &&
    ++			sed "s/^/ /" ../${hash}_committag &&
    ++			echo &&
    ++			echo message
    ++		} >../${hash}_emptymergetag &&
    ++		test_must_fail git hash-object -t commit -w ../${hash}_emptymergetag 2>err &&
    ++		test_grep "Failed to convert object" err
    ++	'
    ++
      	test_expect_success 'rev-parse maps oid of object borrowed from alternate' '
      		for repo in alt borrow
    + 		do
     @@ t/t1016-compatObjectFormat.sh: compare_oids 'blob' more "$more_sha1_oid" "$more_sha256_oid"
      compare_oids 'blob' another "$another_sha1_oid" "$another_sha256_oid"
      compare_oids 'tree' tree2 "$tree2_sha1_oid" "$tree2_sha256_oid"
-- 
2.56.1


  parent reply	other threads:[~2026-11-19 14:18 UTC|newest]

Thread overview: 14+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-12 19:57 [PATCH 0/3] compatObjectFormat: fix a BUG(), wrong id and unknown headers Adrian Ratiu
2026-11-12 19:57 ` [PATCH 1/3] setup: reject compatObjectFormat == objectFormat Adrian Ratiu
2026-11-16  7:36   ` Patrick Steinhardt
2026-11-17 10:42     ` Adrian Ratiu
2026-11-12 19:57 ` [PATCH 2/3] rev-parse: die when --output-object-format cannot map an object Adrian Ratiu
2026-11-16  7:37   ` Patrick Steinhardt
2026-11-17 11:55     ` Adrian Ratiu
2026-11-12 19:57 ` [PATCH 3/3] object-file-convert: copy unknown commit headers unchanged Adrian Ratiu
2026-11-16  7:37   ` Patrick Steinhardt
2026-11-17 15:27     ` Adrian Ratiu
2026-11-19 14:18 ` Adrian Ratiu [this message]
2026-11-19 14:18   ` [PATCH v2 1/3] setup: reject compatObjectFormat == objectFormat Adrian Ratiu
2026-11-19 14:18   ` [PATCH v2 2/3] rev-parse: die when --output-object-format cannot map an object Adrian Ratiu
2026-11-19 14:18   ` [PATCH v2 3/3] object-file-convert: copy unknown commit headers unchanged Adrian Ratiu

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=cover.1794932695.git.adrian.ratiu@collabora.com \
    --to=adrian.ratiu@collabora.com \
    --cc=ebiederm@xmission.com \
    --cc=git@vger.kernel.org \
    --cc=gitster@pobox.com \
    --cc=ps@pks.im \
    --cc=sandals@crustytoothpaste.net \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox