[RFC PATCH 0/4] sign a SHA-256 digest of the tree in commits and tags

[RFC PATCH 0/4] sign a SHA-256 digest of the tree in commits and tags

[Scott Chacon]

I'm concerned about the ecosystem impact of moving the `git init` default
hashing function to SHA-256 in 3.0. I have suggested that it may be more 
feasible with similar benefits to add the ability to inject an independently
calculated and verifiable tree content sha into signed objects instead.

This RFC series is meant to demonstrate how this might work.

It adds the ability to directly rehash the full tree contents when signing a
commit or tag with SHA-256 without the repository needing to be in the sha256
object format.

In this series "git tag -s --hash=sha256" and "git commit -S --hash=sha256" 
compute a SHA-256 digest over every file in the tree (and submodules) and put
that additional hash in a header before signing:

  object 78bd45828aa36fbde3161f49da15272dff3d06f5
  type commit
  tag v1.0
  tagger A U Thor <author@example.com> 1790749714 +0200
  tree-sha256 775aff90d07c9a73f19ef83ab89bd8e95d1d835325d53cc57a2232b015783d89

  Release 1.0
  -----BEGIN SSH SIGNATURE-----

For a commit it goes after "committer", before "gpgsig". Setting
gpg.treeHash=sha256 makes it the default for everything you sign.

The digest is SHA-256 over one record per file, sorted by path:

  <hex sha256 of content> SP <path> NUL

The file mode isn't included. Submodules are followed into their own
repositories and contribute "<hex digest of their tree> SP <path>/ NUL",
so the signature covers their contents too; if a submodule isn't
available, we fail rather than sign something we can't vouch for.

Old versions of Git are fine with the new header: fsck ignores extra
headers after "tagger" by default (and always for commits), and "git
tag -v" and "git verify-commit" check the signature as before.

  - Patch 1 adds the digest, with a test-tool helper so it can be
    tested on its own.
  - Patches 2 and 3 add --hash to "git tag" and "git commit".
  - Patch 4 adds gpg.treeHash.

From a speed perspective, it's not fast but it's not slow. The default
build on my M5 is 245ms for a git.git signed tag call, ~5s for the Linux
tree. An accelerated OpenSSL build is 135ms for git.git, 1.8s for Linux.

However, this is single threaded. We could easily do parallel hashing which
should make it many times faster - my previous tests in Rust on 18 threads
on my M5 did git.git in 36ms and Linux tree in 0.5s (verified the same hash).

Not in this series, and what I'd like opinions on:

  - Any interest? Would the list find this approach a viable alternative
    to not switching the default hash function to sha-256 in 3.0? Not that
    it wouldn't be an available object format, but that it wouldn't need to
    be the default one.

  - Verification. "git tag -v" and "git verify-commit" don't recompute
    the digest yet. I'd like to agree on the format before adding that.

  - Excluding submodules. Large projects can have submodules that most
    people never check out, and they can't sign with --hash today. One
    option is an "excluded:<commit>" header that still covers the
    pinned commit but not its contents, with a header listing the
    excluded paths so that verification can report them.

  - Naming. The header is "tree-sha256", the option "--hash", and the
    config "gpg.treeHash". I'm not attached to any of them.

Scott Chacon (4):
  tree-sha256: hash the contents of a tree with SHA-256
  tag: add --hash=sha256 to sign a tree-sha256 header
  commit: add --hash=sha256 to sign a tree-sha256 header
  gpg: add gpg.treeHash to sign a tree-sha256 header by default

 Documentation/config/gpg.adoc |   6 +
 Documentation/git-commit.adoc |  11 +-
 Documentation/git-tag.adoc    |  11 +-
 Makefile                      |   2 +
 builtin/commit.c              |  46 ++++++-
 builtin/tag.c                 |  41 +++++-
 meson.build                   |   1 +
 t/helper/meson.build          |   1 +
 t/helper/test-tool.c          |   1 +
 t/helper/test-tool.h          |   1 +
 t/helper/test-tree-sha256.c   |  31 +++++
 t/meson.build                 |   2 +
 t/t1018-tree-sha256.sh        | 123 +++++++++++++++++
 t/t7032-tree-sha256-signed.sh | 169 +++++++++++++++++++++++
 tree-sha256.c                 | 247 ++++++++++++++++++++++++++++++++++
 tree-sha256.h                 |  36 +++++
 16 files changed, 720 insertions(+), 9 deletions(-)
 create mode 100644 t/helper/test-tree-sha256.c
 create mode 100755 t/t1018-tree-sha256.sh
 create mode 100755 t/t7032-tree-sha256-signed.sh
 create mode 100644 tree-sha256.c
 create mode 100644 tree-sha256.h


base-commit: a018953688f1b10bddf91bff8747068f5f4746a4
-- 
2.50.1 (Apple Git-155)

[RFC PATCH 1/4] tree-sha256: hash the contents of a tree with SHA-256

[Scott Chacon]

Add a way to compute a SHA-256 digest of the contents of a tree that
doesn't depend on the object format, so that it can be put in the
signed payload. Each blob in the tree, recursively, becomes one record, 
and the digest is SHA-256 over the records sorted by path:

  <hex sha256 of content> SP <path> NUL

A gitlink is followed into the submodule's own repository, and the
tree of the commit it pins is hashed in the same way, giving one record
with a trailing slash on the path:

  <hex digest of submodule tree> SP <path>/ NUL

A path never ends in a slash in a tree, so these can't be confused
with files. If a submodule isn't available, or doesn't have the pinned
commit, we can't say anything about its contents and so fail, listing
every one of them. A submodule that pins a commit already being hashed
above it would be recorded as "cycle:<commit>" instead, which can't
happen without a hash collision but keeps the walk from going around
forever if it does.

The digest can be reproduced with "git ls-tree -r", "git cat-file"
and sha256sum, which is how the new test checks it, through a new
"test-tool tree-sha256". The following patches use it in "git tag"
and "git commit".

---
 Makefile                    |   2 +
 meson.build                 |   1 +
 t/helper/meson.build        |   1 +
 t/helper/test-tool.c        |   1 +
 t/helper/test-tool.h        |   1 +
 t/helper/test-tree-sha256.c |  31 +++++
 t/meson.build               |   1 +
 t/t1018-tree-sha256.sh      | 123 +++++++++++++++++++
 tree-sha256.c               | 238 ++++++++++++++++++++++++++++++++++++
 tree-sha256.h               |  30 +++++
 10 files changed, 429 insertions(+)
 create mode 100644 t/helper/test-tree-sha256.c
 create mode 100755 t/t1018-tree-sha256.sh
 create mode 100644 tree-sha256.c
 create mode 100644 tree-sha256.h

diff --git a/Makefile b/Makefile
index c649c93c51..e042163635 100644
--- a/Makefile
+++ b/Makefile
@@ -878,6 +878,7 @@ TEST_BUILTINS_OBJS += test-submodule.o
 TEST_BUILTINS_OBJS += test-subprocess.o
 TEST_BUILTINS_OBJS += test-synthesize.o
 TEST_BUILTINS_OBJS += test-trace2.o
+TEST_BUILTINS_OBJS += test-tree-sha256.o
 TEST_BUILTINS_OBJS += test-truncate.o
 TEST_BUILTINS_OBJS += test-userdiff.o
 TEST_BUILTINS_OBJS += test-wildmatch.o
@@ -1357,6 +1358,7 @@ LIB_OBJS += trailer.o
 LIB_OBJS += transport-helper.o
 LIB_OBJS += transport.o
 LIB_OBJS += tree-diff.o
+LIB_OBJS += tree-sha256.o
 LIB_OBJS += tree-walk.o
 LIB_OBJS += tree.o
 LIB_OBJS += unpack-trees.o
diff --git a/meson.build b/meson.build
index 0a95d90d21..771e1247f7 100644
--- a/meson.build
+++ b/meson.build
@@ -562,6 +562,7 @@ libgit_sources = [
   'transport-helper.c',
   'transport.c',
   'tree-diff.c',
+  'tree-sha256.c',
   'tree-walk.c',
   'tree.c',
   'unpack-trees.c',
diff --git a/t/helper/meson.build b/t/helper/meson.build
index 3235f10ab8..6f6a4e0a42 100644
--- a/t/helper/meson.build
+++ b/t/helper/meson.build
@@ -72,6 +72,7 @@ test_tool_sources = [
   'test-synthesize.c',
   'test-tool.c',
   'test-trace2.c',
+  'test-tree-sha256.c',
   'test-truncate.c',
   'test-userdiff.c',
   'test-wildmatch.c',
diff --git a/t/helper/test-tool.c b/t/helper/test-tool.c
index b71a22b43b..d3452f7297 100644
--- a/t/helper/test-tool.c
+++ b/t/helper/test-tool.c
@@ -84,6 +84,7 @@ static struct test_cmd cmds[] = {
 	{ "subprocess", cmd__subprocess },
 	{ "synthesize", cmd__synthesize },
 	{ "trace2", cmd__trace2 },
+	{ "tree-sha256", cmd__tree_sha256 },
 	{ "truncate", cmd__truncate },
 	{ "userdiff", cmd__userdiff },
 	{ "xml-encode", cmd__xml_encode },
diff --git a/t/helper/test-tool.h b/t/helper/test-tool.h
index f2885b33d5..410d71f6a9 100644
--- a/t/helper/test-tool.h
+++ b/t/helper/test-tool.h
@@ -77,6 +77,7 @@ int cmd__submodule_nested_repo_config(int argc, const char **argv);
 int cmd__subprocess(int argc, const char **argv);
 int cmd__synthesize(int argc, const char **argv);
 int cmd__trace2(int argc, const char **argv);
+int cmd__tree_sha256(int argc, const char **argv);
 int cmd__truncate(int argc, const char **argv);
 int cmd__userdiff(int argc, const char **argv);
 int cmd__xml_encode(int argc, const char **argv);
diff --git a/t/helper/test-tree-sha256.c b/t/helper/test-tree-sha256.c
new file mode 100644
index 0000000000..d3fca5c0b7
--- /dev/null
+++ b/t/helper/test-tree-sha256.c
@@ -0,0 +1,31 @@
+#define USE_THE_REPOSITORY_VARIABLE
+
+#include "test-tool.h"
+#include "git-compat-util.h"
+#include "hash.h"
+#include "object-name.h"
+#include "repository.h"
+#include "setup.h"
+#include "strbuf.h"
+#include "tree-sha256.h"
+
+int cmd__tree_sha256(int argc, const char **argv)
+{
+	struct object_id oid;
+	struct strbuf hex = STRBUF_INIT;
+	int ret = 0;
+
+	setup_git_directory(the_repository);
+	if (argc != 2)
+		die("usage: test-tool tree-sha256 <tree-ish>");
+	if (repo_get_oid(the_repository, argv[1], &oid))
+		die("not a valid object name: %s", argv[1]);
+
+	if (tree_sha256_hex(the_repository, &oid, &hex))
+		ret = 1;
+	else
+		puts(hex.buf);
+
+	strbuf_release(&hex);
+	return ret;
+}
diff --git a/t/meson.build b/t/meson.build
index 3ca7b27104..8ff3dbe69d 100644
--- a/t/meson.build
+++ b/t/meson.build
@@ -172,6 +172,7 @@ integration_tests = [
   't1015-read-index-unmerged.sh',
   't1016-compatObjectFormat.sh',
   't1017-cat-file-remote-object-info.sh',
+  't1018-tree-sha256.sh',
   't1020-subdirectory.sh',
   't1022-read-tree-partial-clone.sh',
   't1050-large.sh',
diff --git a/t/t1018-tree-sha256.sh b/t/t1018-tree-sha256.sh
new file mode 100755
index 0000000000..ff2edd159e
--- /dev/null
+++ b/t/t1018-tree-sha256.sh
@@ -0,0 +1,123 @@
+#!/bin/sh
+
+test_description='SHA-256 digest of the contents of a tree'
+
+. ./test-lib.sh
+
+# Recompute the tree-sha256 of <rev> in repository <dir> by hand: one
+# "<sha256 of content> <path>" record per file and "<digest> <path>/"
+# per submodule, sorted by path, NUL-terminated and hashed together.
+expect_tree_sha256 () {
+	git -C "$1" ls-tree -r --format="%(objectmode) %(objectname) %(path)" "$2" |
+	while read mode oid path
+	do
+		case "$mode" in
+		160000)
+			printf "%s %s/\n" "$(expect_tree_sha256 "$1/$path" "$oid")" "$path" ;;
+		*)
+			printf "%s %s\n" "$(git -C "$1" cat-file blob "$oid" |
+					    test-tool sha256)" "$path" ;;
+		esac
+	done |
+	LC_ALL=C sort -t " " -k2 |
+	tr "\n" "\000" |
+	test-tool sha256
+}
+
+test_expect_success 'setup' '
+	git config --global protocol.file.allow always &&
+
+	git init inner &&
+	test_commit -C inner inner-file &&
+	git init sub &&
+	test_commit -C sub sub-file &&
+	git -C sub submodule add ../inner inner &&
+	git -C sub commit -m "add inner" &&
+
+	mkdir -p a/deeper &&
+	echo one >a/deeper/file &&
+	echo two >a.b &&
+	echo exe >exe &&
+	git add a a.b exe &&
+	test_ln_s_add a.b link &&
+	git commit -m initial
+'
+
+test_expect_success 'digest of files, directories and symlinks' '
+	expect_tree_sha256 . HEAD >expect &&
+	test-tool tree-sha256 HEAD >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'commits, tags and trees give the same digest' '
+	git tag -a -m tag v1 &&
+	test-tool tree-sha256 v1 >tag &&
+	test-tool tree-sha256 HEAD^{tree} >tree &&
+	test_cmp expect tag &&
+	test_cmp expect tree
+'
+
+test_expect_success 'file mode is not part of the digest' '
+	test_chmod +x exe &&
+	git commit -m executable &&
+	test-tool tree-sha256 HEAD >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'content and paths are' '
+	echo changed >a/deeper/file &&
+	git commit -a -m changed &&
+	test-tool tree-sha256 HEAD >changed &&
+	! test_cmp expect changed &&
+
+	git mv a.b a.c &&
+	git commit -m renamed &&
+	test-tool tree-sha256 HEAD >renamed &&
+	! test_cmp changed renamed
+'
+
+test_expect_success 'submodules are hashed recursively' '
+	git submodule add ./sub sub &&
+	git submodule update --init --recursive &&
+	git commit -m "add sub" &&
+	expect_tree_sha256 . HEAD >expect &&
+	test-tool tree-sha256 HEAD >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'submodule contents are part of the digest' '
+	test_commit -C sub/inner more &&
+	git -C sub commit -a -m "update inner" &&
+	git commit -a -m "update sub" &&
+	test-tool tree-sha256 HEAD >updated &&
+	! test_cmp expect updated &&
+	expect_tree_sha256 . HEAD >expect &&
+	test_cmp expect updated
+'
+
+test_expect_success 'submodules are read from their gitdir without a worktree' '
+	mv sub/inner inner.away &&
+	test_when_finished "mv inner.away sub/inner" &&
+	test-tool tree-sha256 HEAD >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success 'unavailable submodules are an error' '
+	mv sub/inner inner.away &&
+	mv sub/.git/modules/inner inner.git.away &&
+	test_when_finished "mv inner.away sub/inner && mv inner.git.away sub/.git/modules/inner" &&
+	test_must_fail test-tool tree-sha256 HEAD 2>err &&
+	test_grep "sub/inner (not checked out)" err
+'
+
+test_expect_success 'submodule missing the pinned commit is an error' '
+	tree=$(printf "160000 commit %s\tinner\n" $(test_oid deadbeef) |
+	       git -C sub mktree) &&
+	(
+		cd sub &&
+		test_must_fail test-tool tree-sha256 $tree 2>err &&
+		test_grep "inner (checked out, but missing commit $(test_oid deadbeef))" err
+	)
+'
+
+test_done
diff --git a/tree-sha256.c b/tree-sha256.c
new file mode 100644
index 0000000000..fb58962232
--- /dev/null
+++ b/tree-sha256.c
@@ -0,0 +1,238 @@
+#include "git-compat-util.h"
+#include "tree-sha256.h"
+#include "commit.h"
+#include "gettext.h"
+#include "hash.h"
+#include "hex.h"
+#include "object.h"
+#include "odb.h"
+#include "oid-array.h"
+#include "pathspec.h"
+#include "repository.h"
+#include "string-list.h"
+#include "strbuf.h"
+#include "tree.h"
+
+struct record {
+	char *path; /* submodules carry a trailing '/' */
+	struct object_id oid;
+	unsigned submodule:1;
+};
+
+struct collect {
+	struct record *items;
+	size_t nr, alloc;
+};
+
+struct walk {
+	/* "<path> (<reason>)" for each submodule that can't be hashed */
+	struct string_list missing;
+};
+
+static int collect_entry(const struct object_id *oid, struct strbuf *base,
+			 const char *pathname, unsigned mode, void *context)
+{
+	struct collect *c = context;
+	struct record *rec;
+
+	if (S_ISDIR(mode))
+		return READ_TREE_RECURSIVE;
+
+	ALLOC_GROW(c->items, c->nr + 1, c->alloc);
+	rec = &c->items[c->nr++];
+	oidcpy(&rec->oid, oid);
+	rec->submodule = S_ISGITLINK(mode);
+	rec->path = xstrfmt("%.*s%s%s", (int)base->len, base->buf, pathname,
+			    rec->submodule ? "/" : "");
+	return 0;
+}
+
+static int record_cmp(const void *a_, const void *b_)
+{
+	const struct record *a = a_, *b = b_;
+	return strcmp(a->path, b->path);
+}
+
+static int hash_tree(struct repository *r, const struct object_id *oid,
+		     const char *prefix, struct oid_array *chain,
+		     struct walk *walk, unsigned char *digest);
+
+static int in_chain(const struct oid_array *chain, const struct object_id *oid)
+{
+	for (size_t i = 0; i < chain->nr; i++)
+		if (oideq(&chain->oid[i], oid))
+			return 1;
+	return 0;
+}
+
+/*
+ * Hash the submodule of "r" at "path", pinned at "commit", and append
+ * its digest in hex to "out". "treeish" is the tree "path" was found
+ * in, which is where .gitmodules is read from if the submodule's
+ * gitdir isn't at "path". "full" is the path from the top repository,
+ * for messages.
+ *
+ * Returns 1 if the submodule is unavailable (recording why in
+ * walk->missing), -1 on other errors and 0 on success.
+ */
+static int hash_submodule(struct repository *r, const struct object_id *treeish,
+			  const char *path, const char *full,
+			  const struct object_id *commit,
+			  struct oid_array *chain, struct walk *walk,
+			  struct strbuf *out)
+{
+	const struct git_hash_algo *sha256 = &hash_algos[GIT_HASH_SHA256];
+	unsigned char digest[GIT_MAX_RAWSZ];
+	struct repository sub;
+	struct strbuf sub_prefix = STRBUF_INIT;
+	int ret;
+
+	if (repo_submodule_init(&sub, r, path, treeish)) {
+		string_list_append_nodup(&walk->missing, xstrfmt("%s (%s)", full,
+				    _("not checked out")));
+		return 1;
+	}
+	if (!odb_has_object(sub.objects, commit, 0)) {
+		string_list_append_nodup(&walk->missing, xstrfmt("%s (%s %s)", full,
+				    _("checked out, but missing commit"),
+				    oid_to_hex(commit)));
+		repo_clear(&sub);
+		return 1;
+	}
+
+	strbuf_addf(&sub_prefix, "%s/", full);
+	oid_array_append(chain, commit);
+	ret = hash_tree(&sub, commit, sub_prefix.buf, chain, walk, digest);
+	chain->nr--;
+	if (!ret)
+		strbuf_addstr(out, hash_to_hex_algop(digest, sha256));
+
+	strbuf_release(&sub_prefix);
+	repo_clear(&sub);
+	return ret;
+}
+
+/*
+ * Hash one tree of repository "r". "prefix" is the path of "r" from the
+ * top repository (empty, or ending in '/'), and "chain" holds the
+ * commits of the submodules being hashed above this one.
+ */
+static int hash_tree(struct repository *r, const struct object_id *oid,
+		     const char *prefix, struct oid_array *chain,
+		     struct walk *walk, unsigned char *digest)
+{
+	const struct git_hash_algo *sha256 = &hash_algos[GIT_HASH_SHA256];
+	struct git_hash_ctx outer;
+	struct collect c = { 0 };
+	struct pathspec pathspec = { 0 };
+	struct strbuf value = STRBUF_INIT;
+	struct tree *tree;
+	int ret = 0;
+
+	tree = repo_parse_tree_indirect(r, oid);
+	if (!tree)
+		return error(_("unable to read tree for %s in %s"),
+			     oid_to_hex(oid), *prefix ? prefix : ".");
+	if (read_tree(r, tree, &pathspec, collect_entry, &c))
+		return error(_("unable to read tree %s"),
+			     oid_to_hex(&tree->object.oid));
+	QSORT(c.items, c.nr, record_cmp);
+
+	git_hash_init(&outer, sha256);
+	for (size_t i = 0; i < c.nr; i++) {
+		struct record *rec = &c.items[i];
+
+		strbuf_reset(&value);
+		if (!rec->submodule) {
+			struct git_hash_ctx ctx;
+			unsigned char blob_digest[GIT_MAX_RAWSZ];
+			enum object_type type;
+			size_t size;
+			void *data;
+
+			data = odb_read_object(r->objects, &rec->oid, &type, &size);
+			if (!data || type != OBJ_BLOB) {
+				free(data);
+				ret = error(_("unable to read blob %s for %s%s"),
+					    oid_to_hex(&rec->oid), prefix, rec->path);
+				break;
+			}
+			git_hash_init(&ctx, sha256);
+			git_hash_update(&ctx, data, size);
+			git_hash_final(blob_digest, &ctx);
+			free(data);
+			strbuf_addstr(&value, hash_to_hex_algop(blob_digest, sha256));
+		} else if (in_chain(chain, &rec->oid)) {
+			/*
+			 * A commit can't contain itself without a hash
+			 * collision, but don't rely on that to stop.
+			 */
+			strbuf_addf(&value, "cycle:%s", oid_to_hex(&rec->oid));
+		} else {
+			char *path = xstrndup(rec->path, strlen(rec->path) - 1);
+			char *full = xstrfmt("%s%s", prefix, path);
+			int res = hash_submodule(r, &tree->object.oid, path, full, &rec->oid,
+						 chain, walk, &value);
+			free(path);
+			free(full);
+			if (res < 0) {
+				ret = -1;
+				break;
+			}
+		}
+
+		git_hash_update(&outer, value.buf, value.len);
+		git_hash_update(&outer, " ", 1);
+		git_hash_update(&outer, rec->path, strlen(rec->path));
+		git_hash_update(&outer, "", 1);
+	}
+	git_hash_final(digest, &outer);
+
+	for (size_t i = 0; i < c.nr; i++)
+		free(c.items[i].path);
+	free(c.items);
+	strbuf_release(&value);
+	return ret;
+}
+
+int tree_sha256_hex(struct repository *r, const struct object_id *oid,
+		    struct strbuf *hex)
+{
+	const struct git_hash_algo *sha256 = &hash_algos[GIT_HASH_SHA256];
+	unsigned char digest[GIT_MAX_RAWSZ];
+	struct walk walk = { .missing = STRING_LIST_INIT_DUP };
+	struct oid_array chain = OID_ARRAY_INIT;
+	struct commit *top;
+	struct tree *tree;
+	int ret;
+
+	tree = repo_parse_tree_indirect(r, oid);
+	if (!tree)
+		return error(_("cannot compute %s: %s does not point to a tree"),
+			     TREE_SHA256_HEADER, oid_to_hex(oid));
+
+	top = lookup_commit_reference_gently(r, oid, 1);
+	if (top)
+		oid_array_append(&chain, &top->object.oid);
+
+	ret = hash_tree(r, &tree->object.oid, "", &chain, &walk, digest);
+
+	if (!ret && walk.missing.nr) {
+		struct strbuf list = STRBUF_INIT;
+
+		string_list_sort(&walk.missing);
+		for (size_t i = 0; i < walk.missing.nr; i++)
+			strbuf_addf(&list, "\n  %s", walk.missing.items[i].string);
+		ret = error(_("%"PRIuMAX" submodule(s) are not available, and "
+			      "a %s can't be computed without their tree hashes:%s\n"
+			      "Check them out with `git submodule update --init --recursive`."),
+			    (uintmax_t)walk.missing.nr, TREE_SHA256_HEADER, list.buf);
+		strbuf_release(&list);
+	}
+	if (!ret)
+		strbuf_addstr(hex, hash_to_hex_algop(digest, sha256));
+
+	string_list_clear(&walk.missing, 0);
+	oid_array_clear(&chain);
+	return ret;
+}
diff --git a/tree-sha256.h b/tree-sha256.h
new file mode 100644
index 0000000000..6d3e5018aa
--- /dev/null
+++ b/tree-sha256.h
@@ -0,0 +1,30 @@
+#ifndef TREE_SHA256_H
+#define TREE_SHA256_H
+
+struct repository;
+struct object_id;
+struct strbuf;
+
+/* The header that carries the digest in signed commits and tags. */
+#define TREE_SHA256_HEADER "tree-sha256"
+
+/*
+ * Compute the tree-sha256 of the tree reachable from "oid" (a tree,
+ * commit or tag) and append it to "hex" as 64 lowercase hex digits.
+ *
+ * Every blob and symlink in the tree, recursively, contributes one
+ * record "<hex sha256 of content> SP <path> NUL". Every submodule
+ * contributes "<hex tree-sha256 of submodule> SP <path>/ NUL", hashed
+ * from the checked-out submodule's own repository at the commit the
+ * superproject pins; a submodule whose commit is already being hashed
+ * further up the chain is recorded as "cycle:<commit> SP <path>/ NUL".
+ * Records are sorted by path (byte order) and the digest is SHA-256
+ * over their concatenation.
+ *
+ * Returns 0 on success. On failure (for example a submodule that is
+ * not checked out) reports every problem with error() and returns -1.
+ */
+int tree_sha256_hex(struct repository *r, const struct object_id *oid,
+		    struct strbuf *hex);
+
+#endif /* TREE_SHA256_H */
-- 
2.50.1 (Apple Git-155)

[RFC PATCH 2/4] tag: add --hash=sha256 to sign a tree-sha256 header

[Scott Chacon]

Teach "git tag -s" and "git tag -u" a "--hash=sha256" option that
puts the tree-sha256 of the tagged tree in a header after "tagger":

  object <commit>
  type commit
  tag <name>
  tagger <ident>
  tree-sha256 <hex>

Being in the header, it is part of the signed payload, and so the
signature now covers the contents of the tagged tree directly. 

"--hash=sha256" without signing is an error, as there is nothing to
gain from an unsigned digest. It also makes "git tag" create a tag
object, so that it isn't silently dropped when making a lightweight
tag. "--hash=none" is accepted so that a later patch can let it
override a configured default.

---
 Documentation/git-tag.adoc    | 11 ++++-
 builtin/tag.c                 | 29 +++++++++++--
 t/meson.build                 |  1 +
 t/t7032-tree-sha256-signed.sh | 76 +++++++++++++++++++++++++++++++++++
 tree-sha256.c                 |  9 +++++
 tree-sha256.h                 |  6 +++
 6 files changed, 128 insertions(+), 4 deletions(-)
 create mode 100755 t/t7032-tree-sha256-signed.sh

diff --git a/Documentation/git-tag.adoc b/Documentation/git-tag.adoc
index cea3202fdb..8901090a6d 100644
--- a/Documentation/git-tag.adoc
+++ b/Documentation/git-tag.adoc
@@ -9,7 +9,7 @@ git-tag - Create, list, delete or verify tags
 SYNOPSIS
 --------
 [synopsis]
-git tag [-a | -s | -u <key-id>] [-f] [-m <msg> | -F <file>] [-e]
+git tag [-a | -s | -u <key-id>] [--hash=<algorithm>] [-f] [-m <msg> | -F <file>] [-e]
 	[(--trailer <token>[(=|:)<value>])...]
 	<tagname> [<commit> | <object>]
 git tag -d <tagname>...
@@ -84,6 +84,15 @@ OPTIONS
 	`gpg.format` configuration variable. See
 	linkgit:git-config[1].
 
+`--hash=<algorithm>`::
+	When signing, add a `tree-sha256` header holding a SHA-256
+	digest of every file in the tagged object's tree, including the
+	contents of checked-out submodules, so that the signature covers
+	the content directly rather than only its SHA-1 object names.
+	_<algorithm>_ is `sha256`, or `none` (the default).
+	Giving `--hash=sha256` without signing is an error. All
+	submodules must be checked out.
+
 `-f`::
 `--force`::
 	Replace an existing tag with the given name (instead of failing)
diff --git a/builtin/tag.c b/builtin/tag.c
index 06c125b53c..9bc4c946d1 100644
--- a/builtin/tag.c
+++ b/builtin/tag.c
@@ -33,9 +33,10 @@
 #include "write-or-die.h"
 #include "object-file-convert.h"
 #include "trailer.h"
+#include "tree-sha256.h"
 
 static const char * const git_tag_usage[] = {
-	N_("git tag [-a | -s | -u <key-id>] [-f] [-m <msg> | -F <file>] [-e]\n"
+	N_("git tag [-a | -s | -u <key-id>] [--hash=<algorithm>] [-f] [-m <msg> | -F <file>] [-e]\n"
 	   "        [(--trailer <token>[(=|:)<value>])...]\n"
 	   "        <tagname> [<commit> | <object>]"),
 	N_("git tag -d <tagname>..."),
@@ -281,6 +282,7 @@ struct create_tag_options {
 	unsigned int message_given:1;
 	unsigned int use_editor:1;
 	unsigned int sign;
+	unsigned int tree_hash;
 	enum {
 		CLEANUP_NONE,
 		CLEANUP_SPACE,
@@ -316,11 +318,19 @@ static void create_tag(const struct object_id *object, const char *object_ref,
 		    "object %s\n"
 		    "type %s\n"
 		    "tag %s\n"
-		    "tagger %s\n\n",
+		    "tagger %s\n",
 		    oid_to_hex(object),
 		    type_name(type),
 		    tag,
 		    git_committer_info(IDENT_STRICT));
+	if (opt->sign && opt->tree_hash) {
+		strbuf_addstr(&header, TREE_SHA256_HEADER " ");
+		if (tree_sha256_hex(the_repository, object, &header))
+			die(_("unable to compute %s for %s"),
+			    TREE_SHA256_HEADER, object_ref);
+		strbuf_addch(&header, '\n');
+	}
+	strbuf_addch(&header, '\n');
 
 	should_edit = opt->use_editor || !opt->message_given;
 	if (should_edit || trailer_args->nr) {
@@ -468,6 +478,7 @@ int cmd_tag(int argc,
 	int cmdmode = 0, create_tag_object = 0;
 	char *msgfile = NULL;
 	const char *keyid = NULL;
+	const char *hash_arg = NULL;
 	struct msg_arg msg = { .buf = STRBUF_INIT };
 	struct ref_transaction *transaction;
 	struct strbuf err = STRBUF_INIT;
@@ -503,6 +514,8 @@ int cmd_tag(int argc,
 			   N_("add custom trailer(s)")),
 		OPT_BOOL('e', "edit", &edit_flag, N_("force edit of tag message")),
 		OPT_BOOL('s', "sign", &opt.sign, N_("annotated and GPG-signed tag")),
+		OPT_STRING(0, "hash", &hash_arg, N_("algorithm"),
+			   N_("sign a tree-sha256 header of the tagged tree (sha256 or none)")),
 		OPT_CLEANUP(&cleanup_arg),
 		OPT_STRING('u', "local-user", &keyid, N_("key-id"),
 					N_("use another key to sign the tag")),
@@ -556,6 +569,14 @@ int cmd_tag(int argc,
 
 	argc = parse_options(argc, argv, prefix, options, git_tag_usage, 0);
 
+	if (hash_arg) {
+		int tree_hash = parse_signing_hash(hash_arg);
+		if (tree_hash < 0)
+			die(_("unsupported --hash value '%s' (use 'sha256' or 'none')"),
+			    hash_arg);
+		opt.tree_hash = tree_hash;
+	}
+
 	if (!cmdmode) {
 		if (argc == 0)
 			cmdmode = 'l';
@@ -579,7 +600,7 @@ int cmd_tag(int argc,
 		set_signing_key(keyid);
 	}
 	create_tag_object = (opt.sign || annotate || msg.given || msgfile ||
-			     edit_flag || trailer_args.nr);
+			     edit_flag || trailer_args.nr || opt.tree_hash);
 
 	if ((create_tag_object || force) && (cmdmode != 0))
 		usage_with_options(git_tag_usage, options);
@@ -683,6 +704,8 @@ int cmd_tag(int argc,
 	if (create_tag_object) {
 		if (force_sign_annotate && !annotate)
 			opt.sign = 1;
+		if (opt.tree_hash && !opt.sign)
+			die(_("--hash=%s requires a signed tag (-s or -u)"), hash_arg);
 		path = repo_git_path(the_repository, "TAG_EDITMSG");
 		create_tag(&object, object_ref, tag, &buf, &opt, &prev, &object,
 			   &trailer_args, path);
diff --git a/t/meson.build b/t/meson.build
index 8ff3dbe69d..ff83368847 100644
--- a/t/meson.build
+++ b/t/meson.build
@@ -877,6 +877,7 @@ integration_tests = [
   't7012-skip-worktree-writing.sh',
   't7030-verify-tag.sh',
   't7031-verify-tag-signed-ssh.sh',
+  't7032-tree-sha256-signed.sh',
   't7060-wtstatus.sh',
   't7061-wtstatus-ignore.sh',
   't7062-wtstatus-ignorecase.sh',
diff --git a/t/t7032-tree-sha256-signed.sh b/t/t7032-tree-sha256-signed.sh
new file mode 100755
index 0000000000..083f25e665
--- /dev/null
+++ b/t/t7032-tree-sha256-signed.sh
@@ -0,0 +1,76 @@
+#!/bin/sh
+
+test_description='signed tags and commits with a tree-sha256 header'
+GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main
+export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME
+
+. ./test-lib.sh
+GNUPGHOME_NOT_USED=$GNUPGHOME
+. "$TEST_DIRECTORY/lib-gpg.sh"
+
+# Print the value of the tree-sha256 header of <type> <object>, if any.
+header_of () {
+	git cat-file "$1" "$2" >object &&
+	sed -n "/^$/q; s/^tree-sha256 //p" object
+}
+
+test_expect_success GPGSSH 'setup' '
+	git config --global gpg.format ssh &&
+	git config --global gpg.ssh.allowedSignersFile "${GPGSSH_ALLOWED_SIGNERS}" &&
+	git config --global user.signingkey "${GPGSSH_KEY_PRIMARY}" &&
+	mkdir dir &&
+	echo one >dir/file &&
+	echo two >file &&
+	git add dir file &&
+	test_tick &&
+	git commit -m initial &&
+	test-tool tree-sha256 HEAD >expect
+'
+
+test_expect_success GPGSSH 'tag -s --hash=sha256 signs a tree-sha256 header' '
+	git tag -s --hash=sha256 -m release v1 &&
+	header_of tag v1 >actual &&
+	test_cmp expect actual &&
+	sed -n "4,5p" object >lines &&
+	test_grep "^tagger " lines &&
+	test_grep "^tree-sha256 " lines &&
+	git tag -v v1 &&
+	git fsck --strict
+'
+
+test_expect_success GPGSSH 'tag -u --hash=sha256 signs a tree-sha256 header' '
+	git tag -u "${GPGSSH_KEY_PRIMARY}" --hash=sha256 -m release v2 &&
+	header_of tag v2 >actual &&
+	test_cmp expect actual &&
+	git tag -v v2
+'
+
+test_expect_success GPGSSH 'tag -s without --hash has no header' '
+	git tag -s -m release v3 &&
+	header_of tag v3 >actual &&
+	test_must_be_empty actual &&
+	git tag -s --hash=none -m release v4 &&
+	header_of tag v4 >actual &&
+	test_must_be_empty actual
+'
+
+test_expect_success GPGSSH 'tag --hash=sha256 requires signing' '
+	test_must_fail git tag --hash=sha256 -m release v5 2>err &&
+	test_grep "requires a signed tag" err &&
+	test_must_fail git tag --hash=sha256 v5 2>err &&
+	test_grep "requires a signed tag" err &&
+	test_must_fail git rev-parse --verify v5
+'
+
+test_expect_success GPGSSH 'tag --hash rejects unknown algorithms' '
+	test_must_fail git tag -s --hash=md5 -m release v5 2>err &&
+	test_grep "unsupported --hash value" err &&
+	test_must_fail git rev-parse --verify v5
+'
+
+test_expect_success GPGSSH 'tag --hash=sha256 needs an object with a tree' '
+	test_must_fail git tag -s --hash=sha256 -m blob v5 HEAD:file &&
+	test_must_fail git rev-parse --verify v5
+'
+
+test_done
diff --git a/tree-sha256.c b/tree-sha256.c
index fb58962232..90f0306521 100644
--- a/tree-sha256.c
+++ b/tree-sha256.c
@@ -236,3 +236,12 @@ int tree_sha256_hex(struct repository *r, const struct object_id *oid,
 	oid_array_clear(&chain);
 	return ret;
 }
+
+int parse_signing_hash(const char *value)
+{
+	if (!strcasecmp(value, "sha256"))
+		return 1;
+	if (!strcasecmp(value, "none"))
+		return 0;
+	return -1;
+}
diff --git a/tree-sha256.h b/tree-sha256.h
index 6d3e5018aa..dc070129ea 100644
--- a/tree-sha256.h
+++ b/tree-sha256.h
@@ -27,4 +27,10 @@ struct strbuf;
 int tree_sha256_hex(struct repository *r, const struct object_id *oid,
 		    struct strbuf *hex);
 
+/*
+ * Parse the value of a --hash=<algorithm> option. Returns 1 for
+ * "sha256", 0 for "none", and -1 for anything else.
+ */
+int parse_signing_hash(const char *value);
+
 #endif /* TREE_SHA256_H */
-- 
2.50.1 (Apple Git-155)

[RFC PATCH 3/4] commit: add --hash=sha256 to sign a tree-sha256 header

[Scott Chacon]

Teach "git commit -S" the same "--hash=sha256" option as "git tag",
which adds the tree-sha256 of the tree being committed as an extra
header after "committer":

  tree <tree>
  parent <parent>
  author <ident>
  committer <ident>
  tree-sha256 <hex>
  gpgsig <signature>

Extra headers are written before the commit is signed, so the
signature covers it, and "git verify-commit" works as before.

When amending, we normally carry over the extra headers of the commit
being amended. Don't do that for tree-sha256, which would be wrong as
soon as the tree changes, and add a new one only if the amended commit
is signed with "--hash=sha256".

---
 Documentation/git-commit.adoc | 11 +++++++-
 builtin/commit.c              | 38 ++++++++++++++++++++++++---
 t/t7032-tree-sha256-signed.sh | 49 +++++++++++++++++++++++++++++++++++
 3 files changed, 93 insertions(+), 5 deletions(-)

diff --git a/Documentation/git-commit.adoc b/Documentation/git-commit.adoc
index 8329c1034b..c027de2adb 100644
--- a/Documentation/git-commit.adoc
+++ b/Documentation/git-commit.adoc
@@ -15,7 +15,7 @@ git commit [-a | --interactive | --patch] [-s] [-v] [-u[<mode>]] [--amend]
 	   [--date=<date>] [--cleanup=<mode>] [--[no-]status]
 	   [-i | -o] [--pathspec-from-file=<file> [--pathspec-file-nul]]
 	   [(--trailer <token>[(=|:)<value>])...] [-S[<keyid>]]
-	   [--] [<pathspec>...]
+	   [--hash=<algorithm>] [--] [<pathspec>...]
 
 DESCRIPTION
 -----------
@@ -400,6 +400,15 @@ changes to tracked files.
 	countermand both `commit.gpgSign` configuration variable, and
 	earlier `--gpg-sign`.
 
+`--hash=<algorithm>`::
+	When signing, add a `tree-sha256` header holding a SHA-256
+	digest of every file in the commit's tree, including the
+	contents of checked-out submodules, so that the signature covers
+	the content directly rather than only its SHA-1 object names.
+	_<algorithm>_ is `sha256`, or `none` (the default).
+	Giving `--hash=sha256` without signing is an error. All
+	submodules must be checked out.
+
 `--`::
 	Do not interpret any more arguments as options.
 
diff --git a/builtin/commit.c b/builtin/commit.c
index 840b6b4083..871a2bdcd7 100644
--- a/builtin/commit.c
+++ b/builtin/commit.c
@@ -43,6 +43,7 @@
 #include "commit-graph.h"
 #include "pretty.h"
 #include "trailer.h"
+#include "tree-sha256.h"
 
 static const char * const builtin_commit_usage[] = {
 	N_("git commit [-a | --interactive | --patch] [-s] [-v] [-u[<mode>]] [--amend]\n"
@@ -52,7 +53,7 @@ static const char * const builtin_commit_usage[] = {
 	   "           [--date=<date>] [--cleanup=<mode>] [--[no-]status]\n"
 	   "           [-i | -o] [--pathspec-from-file=<file> [--pathspec-file-nul]]\n"
 	   "           [(--trailer <token>[(=|:)<value>])...] [-S[<keyid>]]\n"
-	   "           [--] [<pathspec>...]"),
+	   "           [--hash=<algorithm>] [--] [<pathspec>...]"),
 	NULL
 };
 
@@ -129,7 +130,8 @@ static int quiet, verbose, no_verify, allow_empty, dry_run, renew_authorship;
 static int config_commit_verbose = -1; /* unspecified */
 static int no_post_rewrite, allow_empty_message, pathspec_file_nul;
 static const char *untracked_files_arg, *force_date, *ignore_submodule_arg, *ignored_arg;
-static const char *sign_commit, *pathspec_from_file;
+static const char *sign_commit, *pathspec_from_file, *hash_arg;
+static int tree_hash;
 static struct strvec trailer_args = STRVEC_INIT;
 
 /*
@@ -1737,6 +1739,8 @@ int cmd_commit(int argc,
 			.flags = PARSE_OPT_OPTARG,
 			.defval = (intptr_t) "",
 		},
+		OPT_STRING(0, "hash", &hash_arg, N_("algorithm"),
+			   N_("sign a tree-sha256 header of the committed tree (sha256 or none)")),
 		/* end commit message options */
 
 		OPT_GROUP(N_("Commit contents options")),
@@ -1821,6 +1825,14 @@ int cmd_commit(int argc,
 	argc = parse_and_validate_options(argc, argv, builtin_commit_options,
 					  builtin_commit_usage,
 					  prefix, current_head, &s);
+	if (hash_arg) {
+		tree_hash = parse_signing_hash(hash_arg);
+		if (tree_hash < 0)
+			die(_("unsupported --hash value '%s' (use 'sha256' or 'none')"),
+			    hash_arg);
+		if (tree_hash && !sign_commit)
+			die(_("--hash=%s requires a signed commit (-S)"), hash_arg);
+	}
 	if (trailer_args.nr)
 		trailer_config_init();
 
@@ -1928,13 +1940,31 @@ int cmd_commit(int argc,
 	}
 
 	if (amend) {
-		const char *exclude_gpgsig[3] = { "gpgsig", "gpgsig-sha256", NULL };
-		extra = read_commit_extra_headers(current_head, exclude_gpgsig);
+		const char *exclude[4] = {
+			"gpgsig", "gpgsig-sha256", TREE_SHA256_HEADER, NULL
+		};
+		extra = read_commit_extra_headers(current_head, exclude);
 	} else {
 		struct commit_extra_header **tail = &extra;
 		append_merge_tag_headers(parents, &tail);
 	}
 
+	if (sign_commit && tree_hash) {
+		struct commit_extra_header **tail = &extra;
+		struct strbuf hex = STRBUF_INIT;
+
+		if (tree_sha256_hex(the_repository,
+				    &the_repository->index->cache_tree->oid, &hex)) {
+			rollback_index_files();
+			die(_("unable to compute %s"), TREE_SHA256_HEADER);
+		}
+		while (*tail)
+			tail = &(*tail)->next;
+		CALLOC_ARRAY(*tail, 1);
+		(*tail)->key = xstrdup(TREE_SHA256_HEADER);
+		(*tail)->value = strbuf_detach(&hex, &(*tail)->len);
+	}
+
 	if (commit_tree_extended(sb.buf, sb.len, &the_repository->index->cache_tree->oid,
 				 parents, &oid, author_ident.buf, NULL,
 				 sign_commit, extra)) {
diff --git a/t/t7032-tree-sha256-signed.sh b/t/t7032-tree-sha256-signed.sh
index 083f25e665..44c363b5d2 100755
--- a/t/t7032-tree-sha256-signed.sh
+++ b/t/t7032-tree-sha256-signed.sh
@@ -73,4 +73,53 @@ test_expect_success GPGSSH 'tag --hash=sha256 needs an object with a tree' '
 	test_must_fail git rev-parse --verify v5
 '
 
+test_expect_success GPGSSH 'commit -S --hash=sha256 signs a tree-sha256 header' '
+	test_tick &&
+	git commit --allow-empty -S --hash=sha256 -m signed &&
+	header_of commit HEAD >actual &&
+	test_cmp expect actual &&
+	git verify-commit HEAD
+'
+
+test_expect_success GPGSSH 'commit -S without --hash has no header' '
+	test_tick &&
+	git commit --allow-empty -S -m signed &&
+	header_of commit HEAD >actual &&
+	test_must_be_empty actual &&
+	git commit --allow-empty -S --hash=none -m signed &&
+	header_of commit HEAD >actual &&
+	test_must_be_empty actual
+'
+
+test_expect_success GPGSSH 'commit --hash=sha256 requires signing' '
+	git rev-parse HEAD >before &&
+	test_must_fail git commit --allow-empty --hash=sha256 -m unsigned 2>err &&
+	test_grep "requires a signed commit" err &&
+	test_must_fail git commit --allow-empty -S --no-gpg-sign --hash=sha256 \
+		-m unsigned 2>err &&
+	test_grep "requires a signed commit" err &&
+	test_must_fail git commit --allow-empty -S --hash=md5 -m signed 2>err &&
+	test_grep "unsupported --hash value" err &&
+	git rev-parse HEAD >after &&
+	test_cmp before after
+'
+
+test_expect_success GPGSSH 'amending recomputes or drops the header' '
+	git commit --allow-empty -S --hash=sha256 -m signed &&
+	echo changed >dir/file &&
+	git add dir/file &&
+	test_tick &&
+	git commit --amend -S --hash=sha256 -m amended &&
+	test-tool tree-sha256 HEAD >expect-amended &&
+	! test_cmp expect expect-amended &&
+	header_of commit HEAD >actual &&
+	test_cmp expect-amended actual &&
+	git verify-commit HEAD &&
+
+	test_tick &&
+	git commit --amend -m "amended unsigned" &&
+	header_of commit HEAD >actual &&
+	test_must_be_empty actual
+'
+
 test_done
-- 
2.50.1 (Apple Git-155)

[RFC PATCH 4/4] gpg: add gpg.treeHash to sign a tree-sha256 header by default

[Scott Chacon]

Someone who wants their signatures to cover the contents of their
trees wants it for every tag and commit they sign, and shouldn't have
to remember "--hash=sha256" each time, much as "tag.gpgSign" and
"commit.gpgSign" save them from remembering "-s" and "-S".

Add "gpg.treeHash", which "git tag" and "git commit" use as the
default for "--hash". It is a single variable rather than one for
each command, since the reason for wanting it is the same for both.

It only applies to objects that are signed: with it set, unsigned
commits and annotated or lightweight tags are made as before, rather
than failing as an explicit "--hash=sha256" without signing does.
"--hash=none" overrides it.

---
 Documentation/config/gpg.adoc |  6 +++++
 Documentation/git-commit.adoc |  2 +-
 Documentation/git-tag.adoc    |  2 +-
 builtin/commit.c              |  8 +++++++
 builtin/tag.c                 | 14 ++++++++++-
 t/t7032-tree-sha256-signed.sh | 44 +++++++++++++++++++++++++++++++++++
 tree-sha256.h                 |  4 ++--
 7 files changed, 75 insertions(+), 5 deletions(-)

diff --git a/Documentation/config/gpg.adoc b/Documentation/config/gpg.adoc
index 240e46c050..6728c13a62 100644
--- a/Documentation/config/gpg.adoc
+++ b/Documentation/config/gpg.adoc
@@ -16,6 +16,12 @@ gpg.format::
 See linkgit:gitformat-signature[5] for the signature format, which differs
 based on the selected `gpg.format`.
 
+gpg.treeHash::
+	When set to `sha256`, `git commit` and `git tag` add a
+	`tree-sha256` header to every commit and tag they sign, as if
+	`--hash=sha256` were given. Defaults to `none`. See the
+	`--hash` option in linkgit:git-commit[1] and linkgit:git-tag[1].
+
 gpg.<format>.program::
 	Use this to customize the program used for the signing format you
 	chose. (see `gpg.program` and `gpg.format`) `gpg.program` can still
diff --git a/Documentation/git-commit.adoc b/Documentation/git-commit.adoc
index c027de2adb..1ed6635eee 100644
--- a/Documentation/git-commit.adoc
+++ b/Documentation/git-commit.adoc
@@ -405,7 +405,7 @@ changes to tracked files.
 	digest of every file in the commit's tree, including the
 	contents of checked-out submodules, so that the signature covers
 	the content directly rather than only its SHA-1 object names.
-	_<algorithm>_ is `sha256`, or `none` (the default).
+	_<algorithm>_ is `sha256`, or `none` to override `gpg.treeHash`.
 	Giving `--hash=sha256` without signing is an error. All
 	submodules must be checked out.
 
diff --git a/Documentation/git-tag.adoc b/Documentation/git-tag.adoc
index 8901090a6d..445be41db1 100644
--- a/Documentation/git-tag.adoc
+++ b/Documentation/git-tag.adoc
@@ -89,7 +89,7 @@ OPTIONS
 	digest of every file in the tagged object's tree, including the
 	contents of checked-out submodules, so that the signature covers
 	the content directly rather than only its SHA-1 object names.
-	_<algorithm>_ is `sha256`, or `none` (the default).
+	_<algorithm>_ is `sha256`, or `none` to override `gpg.treeHash`.
 	Giving `--hash=sha256` without signing is an error. All
 	submodules must be checked out.
 
diff --git a/builtin/commit.c b/builtin/commit.c
index 871a2bdcd7..7e56c434db 100644
--- a/builtin/commit.c
+++ b/builtin/commit.c
@@ -1687,6 +1687,14 @@ static int git_commit_config(const char *k, const char *v,
 		sign_commit = git_config_bool(k, v) ? "" : NULL;
 		return 0;
 	}
+	if (!strcmp(k, "gpg.treehash")) {
+		if (!v)
+			return config_error_nonbool(k);
+		tree_hash = parse_signing_hash(v);
+		if (tree_hash < 0)
+			return error(_("invalid value for '%s': '%s'"), k, v);
+		return 0;
+	}
 	if (!strcmp(k, "commit.verbose")) {
 		int is_bool;
 		config_commit_verbose = git_config_bool_or_int(k, v, ctx->kvi,
diff --git a/builtin/tag.c b/builtin/tag.c
index 9bc4c946d1..86871317ed 100644
--- a/builtin/tag.c
+++ b/builtin/tag.c
@@ -51,6 +51,7 @@ static const char * const git_tag_usage[] = {
 static unsigned int colopts;
 static int force_sign_annotate;
 static int config_sign_tag = -1; /* unspecified */
+static int config_tree_hash;
 
 static int list_tags(struct ref_filter *filter, struct ref_sorting *sorting,
 		     struct ref_format *format)
@@ -223,6 +224,15 @@ static int git_tag_config(const char *var, const char *value,
 		return 0;
 	}
 
+	if (!strcmp(var, "gpg.treehash")) {
+		if (!value)
+			return config_error_nonbool(var);
+		config_tree_hash = parse_signing_hash(value);
+		if (config_tree_hash < 0)
+			return error(_("invalid value for '%s': '%s'"), var, value);
+		return 0;
+	}
+
 	if (!strcmp(var, "tag.forcesignannotated")) {
 		force_sign_annotate = git_config_bool(var, value);
 		return 0;
@@ -601,6 +611,8 @@ int cmd_tag(int argc,
 	}
 	create_tag_object = (opt.sign || annotate || msg.given || msgfile ||
 			     edit_flag || trailer_args.nr || opt.tree_hash);
+	if (!hash_arg)
+		opt.tree_hash = config_tree_hash;
 
 	if ((create_tag_object || force) && (cmdmode != 0))
 		usage_with_options(git_tag_usage, options);
@@ -704,7 +716,7 @@ int cmd_tag(int argc,
 	if (create_tag_object) {
 		if (force_sign_annotate && !annotate)
 			opt.sign = 1;
-		if (opt.tree_hash && !opt.sign)
+		if (opt.tree_hash && !opt.sign && hash_arg)
 			die(_("--hash=%s requires a signed tag (-s or -u)"), hash_arg);
 		path = repo_git_path(the_repository, "TAG_EDITMSG");
 		create_tag(&object, object_ref, tag, &buf, &opt, &prev, &object,
diff --git a/t/t7032-tree-sha256-signed.sh b/t/t7032-tree-sha256-signed.sh
index 44c363b5d2..5a656a7816 100755
--- a/t/t7032-tree-sha256-signed.sh
+++ b/t/t7032-tree-sha256-signed.sh
@@ -122,4 +122,48 @@ test_expect_success GPGSSH 'amending recomputes or drops the header' '
 	test_must_be_empty actual
 '
 
+test_expect_success GPGSSH 'gpg.treeHash signs the header by default' '
+	test-tool tree-sha256 HEAD >expect &&
+	test_config gpg.treeHash sha256 &&
+	git tag -s -m release v6 &&
+	header_of tag v6 >actual &&
+	test_cmp expect actual &&
+	test_tick &&
+	git commit --allow-empty -S -m signed &&
+	header_of commit HEAD >actual &&
+	test_cmp expect actual
+'
+
+test_expect_success GPGSSH 'gpg.treeHash leaves unsigned objects alone' '
+	test_config gpg.treeHash sha256 &&
+	git tag -a -m annotated v7 &&
+	header_of tag v7 >actual &&
+	test_must_be_empty actual &&
+	git tag v8 &&
+	test "$(git cat-file -t v8)" = commit &&
+	test_tick &&
+	git commit --allow-empty -m unsigned &&
+	header_of commit HEAD >actual &&
+	test_must_be_empty actual
+'
+
+test_expect_success GPGSSH '--hash=none overrides gpg.treeHash' '
+	test_config gpg.treeHash sha256 &&
+	git tag -s --hash=none -m release v9 &&
+	header_of tag v9 >actual &&
+	test_must_be_empty actual &&
+	test_tick &&
+	git commit --allow-empty -S --hash=none -m signed &&
+	header_of commit HEAD >actual &&
+	test_must_be_empty actual
+'
+
+test_expect_success GPGSSH 'invalid gpg.treeHash is an error' '
+	test_config gpg.treeHash md5 &&
+	test_must_fail git tag -s -m release v10 2>err &&
+	test_grep "invalid value for .gpg.treehash." err &&
+	test_must_fail git commit --allow-empty -S -m signed 2>err &&
+	test_grep "invalid value for .gpg.treehash." err
+'
+
 test_done
diff --git a/tree-sha256.h b/tree-sha256.h
index dc070129ea..6d54c2c9f9 100644
--- a/tree-sha256.h
+++ b/tree-sha256.h
@@ -28,8 +28,8 @@ int tree_sha256_hex(struct repository *r, const struct object_id *oid,
 		    struct strbuf *hex);
 
 /*
- * Parse the value of a --hash=<algorithm> option. Returns 1 for
- * "sha256", 0 for "none", and -1 for anything else.
+ * Parse the value of a --hash=<algorithm> option or of gpg.treeHash.
+ * Returns 1 for "sha256", 0 for "none", and -1 for anything else.
  */
 int parse_signing_hash(const char *value);
 
-- 
2.50.1 (Apple Git-155)

Re: [RFC PATCH 1/4] tree-sha256: hash the contents of a tree with SHA-256

[Junio C Hamano]

Scott Chacon <scott@gitbutler.net> writes:

> Add a way to compute a SHA-256 digest of the contents of a tree that
> doesn't depend on the object format, so that it can be put in the
> signed payload. Each blob in the tree, recursively, becomes one record, 
> and the digest is SHA-256 over the records sorted by path:
>
>   <hex sha256 of content> SP <path> NUL

Would three trees, one records a blob with a single word "hello" at
a path as an executable regular file, another records the same blob
at the same path but as a non-executable regular file, and the third
records a symbolic link whose target is "hello", hash to the same
result?  Should they?

> +static int hash_tree(struct repository *r, const struct object_id *oid,
> +		     const char *prefix, struct oid_array *chain,
> +		     struct walk *walk, unsigned char *digest)
> +{
> +	const struct git_hash_algo *sha256 = &hash_algos[GIT_HASH_SHA256];
> +	struct git_hash_ctx outer;
> +	struct collect c = { 0 };
> +	struct pathspec pathspec = { 0 };
> +	struct strbuf value = STRBUF_INIT;
> +	struct tree *tree;
> +	int ret = 0;
> +
> +	tree = repo_parse_tree_indirect(r, oid);
> +	if (!tree)
> +		return error(_("unable to read tree for %s in %s"),
> +			     oid_to_hex(oid), *prefix ? prefix : ".");
> +	if (read_tree(r, tree, &pathspec, collect_entry, &c))
> +		return error(_("unable to read tree %s"),
> +			     oid_to_hex(&tree->object.oid));
> +	QSORT(c.items, c.nr, record_cmp);

I am somewhat torn but moderately against this sorting there.  If
we have two tree objects that would result in the same checkout,
but one is corrupt in such a way that whose entries are not sorted
correctly, we want them to hash to a different value to signal that,
don't we?

> +	git_hash_init(&outer, sha256);
> +	for (size_t i = 0; i < c.nr; i++) {
> +		struct record *rec = &c.items[i];
> +
> +		strbuf_reset(&value);
> +		if (!rec->submodule) {
> +			struct git_hash_ctx ctx;
> +			unsigned char blob_digest[GIT_MAX_RAWSZ];
> +			enum object_type type;
> +			size_t size;
> +			void *data;
> +
> +			data = odb_read_object(r->objects, &rec->oid, &type, &size);
> +			if (!data || type != OBJ_BLOB) {
> +				free(data);
> +				ret = error(_("unable to read blob %s for %s%s"),
> +					    oid_to_hex(&rec->oid), prefix, rec->path);
> +				break;
> +			}
> +			git_hash_init(&ctx, sha256);
> +			git_hash_update(&ctx, data, size);
> +			git_hash_final(blob_digest, &ctx);
> +			free(data);
> +			strbuf_addstr(&value, hash_to_hex_algop(blob_digest, sha256));

This forces us to read the inflated blob contents as a whole in-core
before we hash.  I wonder if we can use the streaming interface like
how archive-{tar,zip}.c uses odb_stream_from_object() to read the
contents in smaller chunks?  Instead of writing the contents out
like they do, we would instead hash the bytes here.

Re: [RFC PATCH 2/4] tag: add --hash=sha256 to sign a tree-sha256 header

[Junio C Hamano]

Scott Chacon <scott@gitbutler.net> writes:

> @@ -316,11 +318,19 @@ static void create_tag(const struct object_id *object, const char *object_ref,
>  		    "object %s\n"
>  		    "type %s\n"
>  		    "tag %s\n"
> -		    "tagger %s\n\n",
> +		    "tagger %s\n",
>  		    oid_to_hex(object),
>  		    type_name(type),
>  		    tag,
>  		    git_committer_info(IDENT_STRICT));
> +	if (opt->sign && opt->tree_hash) {
> +		strbuf_addstr(&header, TREE_SHA256_HEADER " ");
> +		if (tree_sha256_hex(the_repository, object, &header))
> +			die(_("unable to compute %s for %s"),
> +			    TREE_SHA256_HEADER, object_ref);
> +		strbuf_addch(&header, '\n');
> +	}
> +	strbuf_addch(&header, '\n');

This is a very nice reorganization.  The hardcoded double LF at the
end was a declaration that we wanted to make it hard to add new
fields, but it becomes a hindrance when we want to add an optional
field.

Re: [RFC PATCH 0/4] sign a SHA-256 digest of the tree in commits and tags

[Junio C Hamano]

Scott Chacon <scott@gitbutler.net> writes:

> I'm concerned about the ecosystem impact of moving the `git init` default
> hashing function to SHA-256 in 3.0. I have suggested that it may be more 
> feasible with similar benefits to add the ability to inject an independently
> calculated and verifiable tree content sha into signed objects instead.
>
> This RFC series is meant to demonstrate how this might work.

I have offered a few minor comments on the implementation, but those
are conditional on the assumption that if this is a good idea, we
would want these improvements.  I have not yet formed an opinion on
the overall direction.

Thanks.

Re: [RFC PATCH 0/4] sign a SHA-256 digest of the tree in commits and tags

[brian m. carlson]

[-- Attachment #1: Type: text/plain, Size: 2932 bytes --]

On 2026-10-02 at 08:18:42, Scott Chacon wrote:
> I'm concerned about the ecosystem impact of moving the `git init` default
> hashing function to SHA-256 in 3.0. I have suggested that it may be more
> feasible with similar benefits to add the ability to inject an independently
> calculated and verifiable tree content sha into signed objects instead.

I don't think this is a good idea.  There are lots of reasons it's not,
but the simplest one is that Git requires collision resistance because
it is impossible to store two different colliding blobs.  We don't have
any such blobs yet, but I fully expect SHA-1 to become as weak as MD5,
in which case there will be a large number of items that cannot be
stored in a Git repository.  Even if you don't want to store those
blobs, there are many people, such as security researchers, who _do_
want to store those blobs and that requires a SHA-256 repository.  Your
approach does nothing to address that problem.

Consequently, we need to make the problem better as soon as possible and
that means moving away from SHA-1.  TLS, OpenPGP, and other major
ecosystems have already made this transition and we're very far behind
the times.  The Canadian government already recommends users to have
moved away from SHA-1 and the U.S. government will no longer allow SHA-1
for any purpose as of 2030.  I want to be clear that 4 years in the
large business and government sector is nothing.

I'll also add that the design we have is the design we've had for many
years and there has been ample opportunity to propose alternative
designs.  The plan for Git 3.0 is around the March timeframe and making
substantial changes now is far too late.  Every major forge has support
for SHA-256, whether publicly or in preview, and no forge has support
for this design, nor do I anticipate it seeing a lot of traction,
especially since we explicitly rejected the kind of half-transition
you're proposing for security and other reasons.  Git 3.0 and the
requirement for SHA-256 were discussed at Git Merge 2024 in Berlin and
discussion has happened on the list quite a bit since then, so it
shouldn't be a surprise to anyone.

The thing you really want is the interoperability work, which can
automatically rewrite repositories from one hash algorithm to another
during a clone or fetch operation.  Yes, it isn't quite that simple for
submodules, but if you recursively clone the repository and all its
submodules, it should be possible to rewrite it in place, although that
hasn't been written yet.  That work has not yet been sent upstream
because some of it was written at $DAYJOB, which requires that we use
Outlook and we all know that Outlook corrupts patches.  However, there
is some intention for another company to handle the polishing and
sending, so it should be available sooner or later.
-- 
brian m. carlson (they/them)
Toronto, Ontario, CA

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 325 bytes --]

Re: [RFC PATCH 0/4] sign a SHA-256 digest of the tree in commits and tags

[Scott Chacon]

Hey all,

There are basically two things to respond to here and that I feel the
list should consider before 3.0.

One is this specific proposal of an independent content hash in a
signed header, which I find interesting and potentially helpful
regarding SHA-1 issues, but not necessarily fundamental.

The other is if the default object store for 3.0 should be sha256 or sha1.

On Fri, Oct 2, 2026 at 9:06 PM brian m. carlson
<sandals@crustytoothpaste.net> wrote:
>
> On 2026-10-02 at 08:18:42, Scott Chacon wrote:
> > I'm concerned about the ecosystem impact of moving the `git init` default
> > hashing function to SHA-256 in 3.0. I have suggested that it may be more
> > feasible with similar benefits to add the ability to inject an independently
> > calculated and verifiable tree content sha into signed objects instead.
>
> I don't think this is a good idea.  There are lots of reasons it's not,
> but the simplest one is that Git requires collision resistance because
> it is impossible to store two different colliding blobs.  We don't have
> any such blobs yet, but I fully expect SHA-1 to become as weak as MD5,
> in which case there will be a large number of items that cannot be
> stored in a Git repository.  Even if you don't want to store those
> blobs, there are many people, such as security researchers, who _do_
> want to store those blobs and that requires a SHA-256 repository.  Your
> approach does nothing to address that problem.

My approach was not meant to address that problem, partially because I
believe it to be an incredibly niche problem. For security researchers
or people over-interpreting NIST guidelines to mean "use at all"
rather than "use for signatures", then SHA256 is clearly already a way
to initiate a Git repository and can be used that way. They can do so
today - making sha256 the default in 3.0 does not help or hinder them.

I don't mean to say we should remove different hash algorithms from
Git, but that the _default_ should not bifurcate the entire community
so that security researchers are slightly happier in still mostly
theoretical situations.

Even if Git were based on MD5 content hashing, nearly everyone using
it in nearly every normal scenario would probably be just fine. We can
sign something that is not based on that hashing function (the basis
of this series), but overall, the social trust mechanisms of pull
sources are the predominant security layer, independent of hashing
function.

It's important to differentiate this, since it's being conflated.
Using SHA-1 for signatures is clearly problematic. Using SHA-1 as the
content-addressing hash for it's Merkle DAG is not. The way Git uses
SHA-1 primarily does not rely on a hash function being collision-free;
it relies on a hash function being one-way, which SHA-1 is perfectly
good for and always will be.

The only real issue is that it _also_ uses that hash for signature
integrity, which means we can solve the main issue simply by not
_also_ using it for signature integrity.

> Consequently, we need to make the problem better as soon as possible and
> that means moving away from SHA-1.  TLS, OpenPGP, and other major
> ecosystems have already made this transition and we're very far behind
> the times.  The Canadian government already recommends users to have
> moved away from SHA-1 and the U.S. government will no longer allow SHA-1
> for any purpose as of 2030.  I want to be clear that 4 years in the
> large business and government sector is nothing.

I feel like this is arguably overstated. This is conflating "any
purpose" with "applying cryptographic protection" / digital
signatures. Part of the point of this series was to ensure that
signing would be based on SHA-256 and could in theory make signatures
on Git objects compliant with these NIST-style mandates while still
using SHA-1 for the more basic odb content-addressing work.

In other words, I don't believe that these governments and businesses
ban SHA-1 _for any purpose_. They ban it for the use of cryptographic
protection and I'm saying that a simpler approach to solving that
problem is to re-seperate content addressing hashes from protective
signature hashes.

TLS, OpenPGP, etc all mostly stopped using SHA-1 for signatures, sure,
but that's because providing security is essentially all those
projects do. Governments still let you use modern browsers and
websockets, even though SHA-1 is used in that protocol, specifically
because it doesn't depend on any security properties of SHA-1.

This series is likewise proposing an alternative, non-SHA-1 based
signing and content verification method along with an argument that
maybe seperating those concerns is simpler and less
backwards-incompatible.

> I'll also add that the design we have is the design we've had for many
> years and there has been ample opportunity to propose alternative
> designs.  The plan for Git 3.0 is around the March timeframe and making
> substantial changes now is far too late.

First of all, if you include the compat work, which imho is incredibly
important to this transition being feasible, "the design that we have"
is not even completed yet and is slightly different every time I hear
it. As recently as 8 months ago, you yourself stated "We don't believe
anyone is getting useful use out of the interoperability code in its
current state" [1] and I can verify that this is still the case -
interop is currently completely unusable.

The point of my tree-sha256 series is to actually massively simplify
the work remaining and user experience impact. I'm saying "don't make
it the default", which means that the entire ecosystem doesn't need to
Y2K everything for the next 5 months. Even in Git core, there are
still _substantial_ changes to make for the compat stuff, if I'm not
mistaken. If pack index v3 isn't in core now, do you think it's going
to be in libgit2 and gix and JGit and whatever by March? Not even the
stuff that landed here 6 years ago is in JGit today.

As for the late hour comment, I've felt that this "flag day" hard cut
has been a rather impractical approach to this problem for a while now
and I have mentioned it to several of you in person in the past.
However, I thought maybe some clever solution would come up over the
last two years, but seeing Emily's talk at Git Merge, this close to
the proposed cutover, convinced me that it's going to be a usability
nightmare for everyone. And as above stated, I'm not convinced that
this is anywhere near valuable enough of an outcome for the cost and
difficulty associated.

I also think that a lot of other people would agree, if they had an
idea that this was coming. I believe that many, many users will be
surprised and confused by this when it hits. It turns out that not
very many people read the mailing list.

> Every major forge has support
> for SHA-256, whether publicly or in preview,

Nobody has access to this for GitHub, which is where almost all usage
is and where the kinks could theoretically have been ironed out. If
3.0 comes out in March, there will have been no time for anyone to
give feedback or make substantial changes before everyone is forced
into real usage of this highly incompatible change.

So Bitbucket doesn't, Gerrit doesn't, GitHub doesn't in any practical
sense (I'm curious if anyone on even this mailing list has access to
it's "preview"). GitLab has it under "experimental". I'm hesitant to
agree that Codeberg or whatever constitutes "every major forge".

If anything, this is one of my biggest problems with this breaking
change proposal - it has not been tested in a real way by nearly
_anyone_, nor are major parts of the transistion plan
(compatObjectFormat, pack index v3, fetch/push compatibility, compat
sig verification, etc) fully implemented even a few months out from
the cutover.

As one small but interesting example, I'm honestly fascinated that
there is only now a thread here about the GitHub specific usability
issues [2] with mixed odb repos (between several GitHub-y people,
nonetheless) that hasn't been previously considered (the "limbo"
idea). This is the kind of thing (among many others, I'm sure) that
would come up if people had time to use this at all before a default
switch.

> and no forge has support
> for this design, nor do I anticipate it seeing a lot of traction,
> especially since we explicitly rejected the kind of half-transition
> you're proposing for security and other reasons.

One of the nice things about the design of this particular series is
that no forge support is needed. It would work today.

The new tag/commit header fscks fine and is transferred fine. You
can't rebase signatures anyhow, so dropped headers aren't an issue
(like commit-ids sometimes are). Verification is trusted locally and
if `verify-commit` and `verify-tag` learn this header too, I'm unclear
what "forge support" you think would be needed. New clients add the
new, more secure header, new clients verify it properly, old clients
fall back gracefully.

> Git 3.0 and the
> requirement for SHA-256 were discussed at Git Merge 2024 in Berlin and
> discussion has happened on the list quite a bit since then, so it
> shouldn't be a surprise to anyone.

Yes, my objection is late-ish, but again, it's because I'm not
satisfied with the transition plan or implementation and I assumed it
would have had more time to have some real world usage before the
cutover.

Furthermore, that's just from someone who has actually been there for
many of these discussions. There are a lot of discussions on the list
that will be a surprise to _users_.

Do you have any idea how many custom scripts (various kinds of hooks,
CI scripts, etc) around the world are going to explode on all new
repositories because they have `/^[0-9a-f]{40}$/` hard coded
somewhere? It will be the first time most users have any idea that Git
3.0 creates repos with a different structure - errors like that or
"fatal: the receiving end does not support this repository's hash
algorithm", or Eclipse simply not working, or a hundred other little
issues, will flood unsuspecting Git users. Furthermore, in many cases
it will be _very_ difficult to figure out why exactly this is
happening on some repos and not others.

So yes, it will surprise many, many people.

> The thing you really want is the interoperability work, which can
> automatically rewrite repositories from one hash algorithm to another
> during a clone or fetch operation.  Yes, it isn't quite that simple for
> submodules, but if you recursively clone the repository and all its
> submodules, it should be possible to rewrite it in place, although that
> hasn't been written yet.  That work has not yet been sent upstream
> because some of it was written at $DAYJOB, which requires that we use
> Outlook and we all know that Outlook corrupts patches.  However, there
> is some intention for another company to handle the polishing and
> sending, so it should be available sooner or later.

I think that well thought through, fully implemented and thoroughly
tested interop work is fundamental and neccesary to a change in the
default object format, yes. I find it confusing, especially for a
project so backwards compatibility focused, that this is not a more
widely held viewpoint.

You don't have to accept a version of this series or it's approach
(though I do believe that some simpler signing strategy change
fundamentally solves the main cryptographic security issues, including
NIST-y gov issues), but if nothing else, I would encourage the group
to ship 3.0 without the SHA-256 default and let it be used more widely
on an opt-in basis, let tools and forges work out the compat issues
and have time to get fixes and modifications upstream, and if it's
still a pressing issue, change the default in 4.0 or whatever.

Thanks,
Scott

[1] https://lore.kernel.org/git/20260207200446.2837699-2-sandals@crustytoothpaste.net/
[2] https://lore.kernel.org/git/20261002224400.GA834158@coredump.intra.peff.net/

Re: [RFC PATCH 0/4] sign a SHA-256 digest of the tree in commits and tags

[Patrick Steinhardt]

On Mon, Oct 05, 2026 at 11:32:54AM +0200, Scott Chacon wrote:
> On Fri, Oct 2, 2026 at 9:06 PM brian m. carlson
> <sandals@crustytoothpaste.net> wrote:
> > On 2026-10-02 at 08:18:42, Scott Chacon wrote:
[snip]
> > Every major forge has support
> > for SHA-256, whether publicly or in preview,
> 
> Nobody has access to this for GitHub, which is where almost all usage
> is and where the kinks could theoretically have been ironed out. If
> 3.0 comes out in March, there will have been no time for anyone to
> give feedback or make substantial changes before everyone is forced
> into real usage of this highly incompatible change.
> 
> So Bitbucket doesn't, Gerrit doesn't, GitHub doesn't in any practical
> sense (I'm curious if anyone on even this mailing list has access to
> it's "preview"). GitLab has it under "experimental". I'm hesitant to
> agree that Codeberg or whatever constitutes "every major forge".
> 
> If anything, this is one of my biggest problems with this breaking
> change proposal - it has not been tested in a real way by nearly
> _anyone_, nor are major parts of the transistion plan
> (compatObjectFormat, pack index v3, fetch/push compatibility, compat
> sig verification, etc) fully implemented even a few months out from
> the cutover.
> 
> As one small but interesting example, I'm honestly fascinated that
> there is only now a thread here about the GitHub specific usability
> issues [2] with mixed odb repos (between several GitHub-y people,
> nonetheless) that hasn't been previously considered (the "limbo"
> idea). This is the kind of thing (among many others, I'm sure) that
> would come up if people had time to use this at all before a default
> switch.

The biggest problem I have is that the ecosystem has been entirely
unwilling to do anything about the SHA-256 move before we announced that
this is going to become mandatory. Only then were developers even able
to convince anybody (especially those paying the wages) to get the time
to implement support for it.

So there is some kind of ossification happening in the space. But things
are finally moving now that the due-date is drawing closer. I would be
extremely hesitant to change course again and drop this breaking change
now that there finally is some movement. Because the only consequence of
that would be that the ecosystem will stop working on it again. And even
more so, I would even expect that this will make the next time we want
to do a breaking change exponentially harder as the lesson learned is
that nobody needs to do anything.

Maybe I'm too pessimistic about this, but I don't think so. We've been
working on this whole transition for almost a decade by now, and only
now where we're forcing the ecosystem to adapt are large players like
GitHub even moving.

Patrick

Re: [RFC PATCH 0/4] sign a SHA-256 digest of the tree in commits and tags

[Scott Chacon]

Thanks Steiny,

A quick response,

On Mon, Oct 5, 2026 at 2:41 PM Patrick Steinhardt <ps@pks.im> wrote:
> The biggest problem I have is that the ecosystem has been entirely
> unwilling to do anything about the SHA-256 move before we announced that
> this is going to become mandatory. Only then were developers even able
> to convince anybody (especially those paying the wages) to get the time
> to implement support for it.

Bit of a simple question, but is it possible that this is because
nobody really finds it a concerning problem?

> So there is some kind of ossification happening in the space. But things
> are finally moving now that the due-date is drawing closer. I would be
> extremely hesitant to change course again and drop this breaking change
> now that there finally is some movement. Because the only consequence of
> that would be that the ecosystem will stop working on it again. And even
> more so, I would even expect that this will make the next time we want
> to do a breaking change exponentially harder as the lesson learned is
> that nobody needs to do anything.
>
> Maybe I'm too pessimistic about this, but I don't think so. We've been
> working on this whole transition for almost a decade by now, and only
> now where we're forcing the ecosystem to adapt are large players like
> GitHub even moving.

I want to remind everyone here quickly what "working on this whole
transition for a decade" has looked like, because this seems to be
phrased like everyone wanted this but GitHub was hesitant and pulled
into this important work only by the heroic 3.0 breaking change
decision.

GitHub has been essentially the _only one_ pushing this endeavour from
the beginning of this problem set.

If we assume Brian, Haggerty, Peff, Taylor and Derrick have been
acting on behalf of GitHub, then you Steiny, are essentially the only
major contributor to this project in the last decade that is not
GitHub/MS (Eric maybe?). Very honestly, nobody else seems to care. GH
has single handedly created this issue and then somehow simultaneously
been the blocking factor to it's rollout because it also,
simultaneously, does not really find it to be an actually important
issue. Google maybe helped design the transition plan in 2017, but
hasn't seemed to care too much since then. Nobody else has really
weighed in, at least with patches.

Scott

Re: [RFC PATCH 0/4] sign a SHA-256 digest of the tree in commits and tags

[brian m. carlson]

[-- Attachment #1: Type: text/plain, Size: 6394 bytes --]

On 2026-10-05 at 14:16:48, Scott Chacon wrote:
> Thanks Steiny,
> 
> A quick response,

Hey,

> On Mon, Oct 5, 2026 at 2:41 PM Patrick Steinhardt <ps@pks.im> wrote:
> > The biggest problem I have is that the ecosystem has been entirely
> > unwilling to do anything about the SHA-256 move before we announced that
> > this is going to become mandatory. Only then were developers even able
> > to convince anybody (especially those paying the wages) to get the time
> > to implement support for it.
> 
> Bit of a simple question, but is it possible that this is because
> nobody really finds it a concerning problem?

I think that's an oversimplification.  I think people don't realize that
Git is using SHA-1 and once SHA-256 is the default they will be very
much in favour of using it.  As I've said elsewhere in the thread, the
need to move away from SHA-1 is going to become gradually urgent for a
large segment of major institutions.

I can say that I've also had inquiries from large government agencies
and corporations and they very much know about SHA-256 and want it.
It's also very much desired by many in the open source community based
on feedback that I've received there.

To respond to what Patrick said, I think in general there is a huge
reluctance to invest in Git as an open source project and much open
source investment is driven by internal corporate needs.  As such,
there's been a huge investment in scaling Git and a lot less investment
in anything else, even if sometimes that ends up with less desirable
outcomes.  Customers get developers paged if their repositories don't
scale, but they don't page about SHA-256.  That doesn't mean it's not
important or valuable.

> > So there is some kind of ossification happening in the space. But things
> > are finally moving now that the due-date is drawing closer. I would be
> > extremely hesitant to change course again and drop this breaking change
> > now that there finally is some movement. Because the only consequence of
> > that would be that the ecosystem will stop working on it again. And even
> > more so, I would even expect that this will make the next time we want
> > to do a breaking change exponentially harder as the lesson learned is
> > that nobody needs to do anything.
> >
> > Maybe I'm too pessimistic about this, but I don't think so. We've been
> > working on this whole transition for almost a decade by now, and only
> > now where we're forcing the ecosystem to adapt are large players like
> > GitHub even moving.
> 
> I want to remind everyone here quickly what "working on this whole
> transition for a decade" has looked like, because this seems to be
> phrased like everyone wanted this but GitHub was hesitant and pulled
> into this important work only by the heroic 3.0 breaking change
> decision.
> 
> GitHub has been essentially the _only one_ pushing this endeavour from
> the beginning of this problem set.

I will merely say in this regard that I don't speak in my corporate
capacity from this email address, so I don't think I'd like to respond
to this statement.  Patrick and I and the other contributors have
discussed SHA-256 and Git 3.0 at the Contributor's Summits in 2024,
2025, and 2026 and so I think there's a good understanding of where
different people and companies have been contributing to that and other
efforts.

What I will say is that my experience on SHA-256 is that it challenges a
lot of assumptions that people have built into their code over the years
and therefore any sort of migration to support SHA-256 involves a lot of
work, including substantial code changes and database migrations.  That
means that sometimes people have been doing substantial work behind the
scenes and it's just not visible until it's done.  You can see how this
works by looking at open source projects like libgit2 and gitoxide,
where extensive changes have landed over time.  My experience is that
reftable is another project where this is the case as well.

> If we assume Brian, Haggerty, Peff, Taylor and Derrick have been
> acting on behalf of GitHub, then you Steiny, are essentially the only
> major contributor to this project in the last decade that is not
> GitHub/MS (Eric maybe?). Very honestly, nobody else seems to care. GH
> has single handedly created this issue and then somehow simultaneously
> been the blocking factor to it's rollout because it also,
> simultaneously, does not really find it to be an actually important
> issue. Google maybe helped design the transition plan in 2017, but
> hasn't seemed to care too much since then. Nobody else has really
> weighed in, at least with patches.

I do want to clarify this, since I think there's a lot of confusion.
When I send contributions or patches from my personal email address,
they're personal contributions.  Only if the patches contain my work
address (which is extremely rarely) are they in my corporate capacity or
done on corporate time.

The SHA-256 work that I've been doing has almost exclusively been in my
personal capacity[0].  There is some of the interoperability work that I
was able to do on work time and those patches reflect the appropriate
email address and sign-off, but before that I have done almost no
SHA-256 work on company time.  This work has been done mostly on nights
and weekends, as with almost all of my other contributions, including on
the security list.  I contribute because I like the project and want it
succeed, not because I'm paid to do so.

I also want to state that I've received a great amount of assistance and
contributions, including reviews, patches, thoughtful ideas, and
miscellaneous assistance, from a wide variety of contributors to the
list and I could not have done it without them.  Someone who has only
provided reviews or design ideas has still aided the SHA-256 project and
Git as a whole immensely.  Patrick is just one of many people who have
aided in such a way.

As mentioned earlier, I am of course not going to comment on anything
related to my employer on any of this.  If you want their opinion, you
should ask them.

[0] The interested reader may wish to run the following command:
    git log --format='%ae' | grep -E '^(sandals|bk2204)@' | sort | uniq -c
-- 
brian m. carlson (they/them)
Toronto, Ontario, CA

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 325 bytes --]

Re: [RFC PATCH 0/4] sign a SHA-256 digest of the tree in commits and tags

[Christian Couder]

On Fri, Oct 2, 2026 at 9:15 PM brian m. carlson
<sandals@crustytoothpaste.net> wrote:

> The thing you really want is the interoperability work, which can
> automatically rewrite repositories from one hash algorithm to another
> during a clone or fetch operation.  Yes, it isn't quite that simple for
> submodules, but if you recursively clone the repository and all its
> submodules, it should be possible to rewrite it in place, although that
> hasn't been written yet.  That work has not yet been sent upstream
> because some of it was written at $DAYJOB, which requires that we use
> Outlook and we all know that Outlook corrupts patches.  However, there
> is some intention for another company to handle the polishing and
> sending, so it should be available sooner or later.

Sorry for the possibly stupid following questions, but I think the
answers might help us get a better idea of what might be needed to get
a smoother transition.

And yeah, I know that many people have said that merging all your
interoperability work should not block Git 3.0. But if it can ensure a
smoother transition, we might want to get at least part of it merged
soon, and the rest in a good shape, anyway.

Is the current state of the work publicly available somewhere? Or
could you make it publicly available somewhere? (Fine if it's only as
patches in a tarball.)

Is the submodule work the only missing part of the interoperability work?

How much work is this? (At one point it seemed to me that it was
around 200 patches.)

If you were to work full time on upstreaming it, how long would you
expect it would take you?

If some of us could help you, how could we best help?

Could you say which company is interested in helping with this? Would
that company be willing to work openly with others on this?

Are there some tests or kinds of automated ways to check that things
work as expected under realistic conditions like:

- using real world repos (large ones, old ones, with submodules, etc),
- mixing a number of new and old clients and servers,
- interacting with other implementations (JGit, libgit2, gitoxide,
forges, CI, etc)?

Thanks for all your work on this in your free time,
Christian.

Re: [RFC PATCH 0/4] sign a SHA-256 digest of the tree in commits and tags

[Johannes Schindelin]

[-- Attachment #1: Type: text/plain, Size: 6051 bytes --]

Hi Scott & Patrick,

On Mon, 5 Oct 2026, Scott Chacon wrote:

> On Mon, Oct 5, 2026 at 2:41 PM Patrick Steinhardt <ps@pks.im> wrote:
>
> > The biggest problem I have is that the ecosystem has been entirely
> > unwilling to do anything about the SHA-256 move before we announced
> > that this is going to become mandatory. Only then were developers even
> > able to convince anybody (especially those paying the wages) to get
> > the time to implement support for it.
> 
> Bit of a simple question, but is it possible that this is because
> nobody really finds it a concerning problem?

I do agree that there has been an enormous reluctance to move to SHA-256.

One part is of course, that there was no sense of urgency, not even with
the SHAttered paper, because of the difficulty to apply it to Git objects
(which by happenstance rather than design made creating collisions
harder).

Out of curiosity, I researched the feasibility half a year ago: generating
two different `.c` files with the same blob OID where one of them carries
_some_ malicious payload (and an enormous number of seemingly random
bytes, carefully ensuring no NULs) would have a rough price tag of ~$10k
and a month of rented RTX 3090s. So that's not _purely_ theoretical, and
those numbers are likely lower today than half a year ago.

From my point of view, though, the much bigger part of the reluctance
stems from the ginormous amount of work required to migrate existing
repositories to SHA-256, and all that for little to no perceived benefit!

And it's not just an incredible amount of work, there are issues:

- Maintaining a local-only SHA-1 <-> SHA-256 mapping would be _required_
  for transition periods (forget about flag days, they are not feasible),
  and the resources (time!) are prohibitive for most serious data shapes.

- There are still no satisfying answers to the question how to deal with
  submodules. "Just use only SHA-1 or only SHA-256" is an answer that I
  heard as frequently as it is out of touch with reality: submodules often
  fetch from 3rd-party projects who don't exactly bend over to do what
  _you_ happen to need.

- There are still only absolutely unsatisfying answers to the question how
  to deal with partial or shallow clones. Unless you try flag days (which
  are, let's face it, practical only for ridiculously small teams).

- It is totally impossible to discern between a short SHA-1 and a short
  SHA-256. (No "this is SHA-256" prefix, or "first letter is an inverse
  hex digit" kind of discerning pattern there.)

  This has many corollaries, e.g.: The minimum hex digits for short OIDs
  changes substantially depending whether or not you have only one hash,
  or maintain a local mapping between SHA-1 <-> SHA-256. Just to name one.

There are many more issues with SHA-256 repositories, not least of which
that many a logic hard-codes "40 hex-digits" as the size of the hash.
Tooling. Services. Platforms. I know that the immediate reaction will be:
"Well, they should have prepared better!" which brings me back to
above-mentioned out-of-touch comment.

The worst part about this? The rationale that we need to switch to SHA-256
by default because SHA-1 makes Git repositories cryptographically weak
rarely matters in practice:

- The Git objects are already on The Server, in most cases. Let's face it,
  Git isn't used in a distributed manner. Most projects have their
  canonical central repository from which everybody clones. It would be
  simply impossible to replace existing objects with SHA-1-same copies on
  those servers.

- While many projects are developed in Git repositories, they are usually
  distributed via packages, including source packages, with separate
  actual cryptographic signatures. And those signatures are what matters,
  not Git's SHA-1.

- The well-known adage that the weakest link in the chain is what breaks
  it is quite true even in code security. It is pretty expensive (see
  above) to create SHA-1 collisions, especially ones that would not be
  spotted _immediately_. It is much, much easier to target the human
  element in the chain. You don't need SHA-1 collisions for that at all,
  just an overworked open source maintainer, and it is also much cheaper.

> > So there is some kind of ossification happening in the space. But
> > things are finally moving now that the due-date is drawing closer. I
> > would be extremely hesitant to change course again and drop this
> > breaking change now that there finally is some movement. Because the
> > only consequence of that would be that the ecosystem will stop working
> > on it again. And even more so, I would even expect that this will make
> > the next time we want to do a breaking change exponentially harder as
> > the lesson learned is that nobody needs to do anything.
> >
> > Maybe I'm too pessimistic about this, but I don't think so. We've been
> > working on this whole transition for almost a decade by now, and only
> > now where we're forcing the ecosystem to adapt are large players like
> > GitHub even moving.

I do agree that essentially only something like the "threat" of Git v3.0
switching to SHA-256 by default could have moved the industry players (and
not all of them, some of them still won't be able to support SHA-256, due
to the lack of funding for the work that would be required).

Having said that, I do think that we have to keep an open mind.

We need to keep the option open to decide "at the last minute" to satisfy
ourselves in Git v3.0 with having excellent support for SHA-256 and at the
same time _not forcing_ everybody and their cats to use it.

A feature like SHA-256 should probably be enabled only because users want
it, not because a few Git contributors want it so badly that they propose
to make it the default in v3.0. _The default_ should be switched to
SHA-256 only to solve a real problem that real users recognize and want to
see solved.

Ciao,
Johannes

Re: [RFC PATCH 0/4] sign a SHA-256 digest of the tree in commits and tags

[Kristoffer Haugsbakk]

On Mon, Oct 5, 2026, at 14:41, Patrick Steinhardt wrote:
> On Mon, Oct 05, 2026 at 11:32:54AM +0200, Scott Chacon wrote:
>> On Fri, Oct 2, 2026 at 9:06 PM brian m. carlson
>> <sandals@crustytoothpaste.net> wrote:
>> > On 2026-10-02 at 08:18:42, Scott Chacon wrote:
> [snip]
>> > Every major forge has support
>> > for SHA-256, whether publicly or in preview,
>>
>> Nobody has access to this for GitHub, which is where almost all usage
>> is and where the kinks could theoretically have been ironed out. If
>> 3.0 comes out in March, there will have been no time for anyone to
>> give feedback or make substantial changes before everyone is forced
>> into real usage of this highly incompatible change.
>>
>>[snip even more]
>>
>> As one small but interesting example, I'm honestly fascinated that
>> there is only now a thread here about the GitHub specific usability
>> issues [2] with mixed odb repos (between several GitHub-y people,
>> nonetheless) that hasn't been previously considered (the "limbo"
>> idea). This is the kind of thing (among many others, I'm sure) that
>> would come up if people had time to use this at all before a default
>> switch.
>
> The biggest problem I have is that the ecosystem has been entirely
> unwilling to do anything about the SHA-256 move before we announced that
> this is going to become mandatory. Only then were developers even able
> to convince anybody (especially those paying the wages) to get the time
> to implement support for it.
>
> So there is some kind of ossification happening in the space. But things
> are finally moving now that the due-date is drawing closer. I would be
> extremely hesitant to change course again and drop this breaking change
> now that there finally is some movement. Because the only consequence of
> that would be that the ecosystem will stop working on it again. And even
> more so, I would even expect that this will make the next time we want
> to do a breaking change exponentially harder as the lesson learned is
> that nobody needs to do anything.
>
> Maybe I'm too pessimistic about this, but I don't think so. We've been
> working on this whole transition for almost a decade by now, and only
> now where we're forcing the ecosystem to adapt are large players like
> GitHub even moving.

The wider ecosystem is one thing. But git(1) itself doesn’t seem
ready at all.

A few days ago, having read Scott Chacon’s blog post and discussions
around it,[1] I wanted to test migrating the Git repo to SHA-256. So how
do I do that? I google around and the most official “transition” program
seems to be this.

https://git-scm.com/docs/hash-function-transition

I.e. a document where you can’t really tell the implementation from the
aspiration.

† 1: It seems he has never linked it here on the list, like in this
     thread.

But okay, the *real* program to convert a repository seems to be

    git fast-export --all

And that was okay. The Git repo seems to have a bit of cruft, and
git-fast-export(1) fails on the first error then suggests a fix so that
you can continue on to the next error. But that’s fine for a one-shot
program. For anyone interested:

    git fast-export --all --reencode=yes --mark-tags \
        --signed-tags=verbatim \
        --tag-of-filtered-object=rewrite >SHA1HERE

Some real loss of fidelity was there though:

1. You can’t for some reason export refs that point to blobs or trees
2. Your Git notes will be effectively lost since they will retain their
   SHA-1 filenames. (This is mentioned in hash-function-transition)

Then you import it with

    git fast-import

But to no one’s surprise (here) this does not work because of the SHA-1
collision submodule.

Okay, dropping that exercise for a second. I would personally be okay
with trying out this migration on my existing repos that are “local
only”. It would clearly be in my interest to find any bugs that are
particular to my workflows. But for that I would that migration where
you keep a mapping of SHA-1 to SHA-256. Or else I will lose Git notes
forever (which I use a lot).

But reading brian’s cousin response:
<asQrWAKQXV9zn1Vq@fruit.crustytoothpaste.net> ... it seems that there is
not enough in git(1) or anywhere else to do that.

So what is anyone outside the group of guts-of-Git developers supposed
to do? It seems we just have to wait until Git 3.0 and see what happens.

And then you can imagine Git 3.0. Someone happens to make a new Git
repository and they use it for three days without trying to push it
somewhere to SHA-1-Only land. But then they do. And the forge has at
least implemented a nice, informative error message. The user only cares
that “a week ago it worked” and “now it is broken”. (“broken”
subjectively.) He feeds it to some oracle and it turns out that they
just need a few commands to convert back to SHA-1, then one command to
turn off the default. Now they have mitigated the “broken Git” problem
for themselves. And what have they lost? It’s not even a hack.

Then zoom out and you might have the wider ecosystem, like forges.
If they don’t implement it in time? Well, maybe that informative error
message becomes:

    remote: unsupported hash algorithm
    remote: if this is your first push of a local repo, here’s
    remote: how to convert your repo to SHA-1:
    remote: <commands>
    remote: and here’s how to turn off this default [...]

Then someone will post that as a question on StackOverflow, get flamed
because the error message “says how to fix it”, get 3000 upvotes, and
the world moves drunkenly on.

The above scenario would be very hyperbolic and too cynical if not for
the context: one person is leading the direct implementation work[2] in
their spare time. In order to migrate Git from a to-be government-wide
banned hash algorithm. That seems like an institutional malfunction.
Somewhere.

To reiterate, there doesn’t seem like there is anything for us
above-average Git-interested users to do yet. So it feels like we
just have to wait until March or a little later, see if the default-
SHA-256 change lands, and see what the fallout is. And that is a bit
frustrating.

† 2: This is to acknowledge that there are other people like reviewers

Re: [RFC PATCH 0/4] sign a SHA-256 digest of the tree in commits and tags

[brian m. carlson]

[-- Attachment #1: Type: text/plain, Size: 5161 bytes --]

On 2026-10-06 at 16:16:07, Kristoffer Haugsbakk wrote:
> And that was okay. The Git repo seems to have a bit of cruft, and
> git-fast-export(1) fails on the first error then suggests a fix so that
> you can continue on to the next error. But that’s fine for a one-shot
> program. For anyone interested:
> 
>     git fast-export --all --reencode=yes --mark-tags \
>         --signed-tags=verbatim \
>         --tag-of-filtered-object=rewrite >SHA1HERE
> 
> Some real loss of fidelity was there though:
> 
> 1. You can’t for some reason export refs that point to blobs or trees
> 2. Your Git notes will be effectively lost since they will retain their
>    SHA-1 filenames. (This is mentioned in hash-function-transition)
> 
> Then you import it with
> 
>     git fast-import
> 
> But to no one’s surprise (here) this does not work because of the SHA-1
> collision submodule.

We actually have support for rewriting submodules in fast-export and
fast-import.  It's a little fussy because you have to rewrite all the
submodules before you rewrite the main repository, but it works.  Here's
a command to handle git.git:

----
#!/bin/sh

temp=$(mktemp -d)
trap 'rm -fr "$temp"' EXIT

GIT_LOCATION="$1"
RESULT="$2"

git -C "$GIT_LOCATION/sha1collisiondetection" fast-export --signed-tags=verbatim --tag-of-filtered-object=drop --export-marks="$temp/sha1dc-sha1.marks" --all >"$temp/sha1dc.export"

git init --bare --object-format=sha256 "$temp/sha1dc"
git -C "$temp/sha1dc" fast-import --export-marks="$temp/sha1dc-sha256.marks" < "$temp/sha1dc.export"

git -C "$GIT_LOCATION" fast-export --reencode=no --signed-tags=verbatim --tag-of-filtered-object=drop --branches --tags >"$temp/git.export"
git init --object-format=sha256 "$RESULT"
git -C "$RESULT" fast-import --rewrite-submodules-from=sha1dc:"$temp/sha1dc-sha1.marks" --rewrite-submodules-to=sha1dc:"$temp/sha1dc-sha256.marks" <"$temp/git.export"
----

And here's an example running it right now (my main branch following
`master` is `dev`):

----
% ./convert-git ~/checkouts/git git-sha256.git
[elided]
% git -C git-sha256.git log -1 --format=oneline dev
05370fd7088edf77bfcd09c8c909450e764a9d10d8304dc333f39f01697c5a84 4th batch for -rc1
----

The downside is that it doesn't produce the same results as the true
interoperability code and it's much slower, and, as I pointed out above,
the user experience is poor.  The advantage is that it's been available
since the original SHA-256 work in about 2.30 or so, so you can totally
make it work almost anywhere.  The above script could also probably be
nicely converted into a generic script that would work on any repository
without too much effort.

> Okay, dropping that exercise for a second. I would personally be okay
> with trying out this migration on my existing repos that are “local
> only”. It would clearly be in my interest to find any bugs that are
> particular to my workflows. But for that I would that migration where
> you keep a mapping of SHA-1 to SHA-256. Or else I will lose Git notes
> forever (which I use a lot).
> 
> But reading brian’s cousin response:
> <asQrWAKQXV9zn1Vq@fruit.crustytoothpaste.net> ... it seems that there is
> not enough in git(1) or anywhere else to do that.

The interoperability work doesn't rewrite notes because it only happens
when cloning or fetching from a repository and notes aren't usually
copied in that case.  In-place rewriting is not yet implemented,
although that's a thing I'd like to work on.  Hooking notes into that
shouldn't be very difficult to do.

The reason more of the interoperability work has not gone upstream is
because the pluggable ODB work has really ended up breaking a lot of
things[0], so sending almost anything requires a bunch of rebasing and
fixing, and I'm presently very burnt out, so I'm doing very little
coding in my free time and doing more cycling, reading, and Factorio:
Space Age.

> The above scenario would be very hyperbolic and too cynical if not for
> the context: one person is leading the direct implementation work[2] in
> their spare time. In order to migrate Git from a to-be government-wide
> banned hash algorithm. That seems like an institutional malfunction.
> Somewhere.

This is the problem with open source, unfortunately.  In the ideal
world, would other people and very especially major companies help out
more?  Sure.  But macOS and FreeBSD also ship one person's bc/dc
implementation as a core part of the OS, there's only one maintainer
each for bash and ncurses, and a lot of other cases.  This is basically
https://xkcd.com/2347/, which, as we all know, is a widespread problem.

As I said elsewhere, everyone is interested in scaling Git to larger and
larger repositories and improving performance, but little else gets
attention.  Those are things I _don't_ really want to work on, which is
why my job is not working on Git.

[0] To be clear, I think it's a great project and I'm very happy to see
the work come in, but it has impacts throughout the codebase.
-- 
brian m. carlson (they/them)
Toronto, Ontario, CA

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 325 bytes --]

Re: [RFC PATCH 0/4] sign a SHA-256 digest of the tree in commits and tags

[brian m. carlson]

[-- Attachment #1: Type: text/plain, Size: 7052 bytes --]

On 2026-10-06 at 09:00:45, Christian Couder wrote:
> On Fri, Oct 2, 2026 at 9:15 PM brian m. carlson
> <sandals@crustytoothpaste.net> wrote:
> 
> > The thing you really want is the interoperability work, which can
> > automatically rewrite repositories from one hash algorithm to another
> > during a clone or fetch operation.  Yes, it isn't quite that simple for
> > submodules, but if you recursively clone the repository and all its
> > submodules, it should be possible to rewrite it in place, although that
> > hasn't been written yet.  That work has not yet been sent upstream
> > because some of it was written at $DAYJOB, which requires that we use
> > Outlook and we all know that Outlook corrupts patches.  However, there
> > is some intention for another company to handle the polishing and
> > sending, so it should be available sooner or later.
> 
> Sorry for the possibly stupid following questions, but I think the
> answers might help us get a better idea of what might be needed to get
> a smoother transition.
> 
> And yeah, I know that many people have said that merging all your
> interoperability work should not block Git 3.0. But if it can ensure a
> smoother transition, we might want to get at least part of it merged
> soon, and the rest in a good shape, anyway.
> 
> Is the current state of the work publicly available somewhere? Or
> could you make it publicly available somewhere? (Fine if it's only as
> patches in a tarball.)

Yeah, it's at https://github.com/bk2204/git.git as `sha256-interop`.

> Is the submodule work the only missing part of the interoperability work?

Not quite.  The limitations are outlined in
https://lore.kernel.org/git/ajCWBG9RHBrm8jMZ@fruit.crustytoothpaste.net/
and in `Documentation/gitformat-hash.adoc` in the branch.

There's no in-place migration tooling, which would be required for
dealing with submodules, since those would need to be migrated before
the main repository.  There are essentially two cases for in-place
migration: adding the other hash as an extra mapping and rewriting all
the objects to the other hash with our hash as a mapping, much like `git
refs migrate` does.

I've started on some of the Rust pieces that I was planning to use for
in-place migration, but someone who wanted to work on a C-based
implementation could also do that.

There's some missing features that I've outlined, like a lack of support
for multi-pack index.  Those can be added, but they're not immediately
essential.  And there are other things which need to be actually
polished and fixed, like the fact that delta resolution is recursive
rather than iterative (which would allow a malicious server to cause
stack exhaustion).  The good news is that most of those things are
labelled with `WIP` and a short description of what needs to be fixed in
the commit message.

One other thing that needs fixing is that we have many pieces of code
that write large numbers of loose objects into the ODB and may randomly
die in places; `git add` is a great example of this.  The object maps
which are used for storing loose objects should ideally be written with
all of those objects at once in a batch and then committed.  However, if
we die at any point, we've written the loose objects, but not the
batched object map, so the repository is then corrupt since it can't
perform mappings.  If we write the objects into the object map one at a
time, then we end up with N object maps and `git gc` runs all the time
to repack those into a smaller set of data.  We therefore need to either
fix the die-die-die behaviour or use ODB transactions to write both
loose objects and the object maps into a temporary directory.  This is a
case where it technically works but it performs awfully, so we do need
to fix it before non-experimental use.

There is a partial rebase of the early entries in the series converted
to use the pluggable ODB work in the `sha256-interop-part-2` branch. The
pluggable ODB work has caused a lot of conflicts in the interop because,
unsurprisingly, both series are intimately involved in the object
database.

> How much work is this? (At one point it seemed to me that it was
> around 200 patches.)

It's presently about 212 patches.

> If you were to work full time on upstreaming it, how long would you
> expect it would take you?

Probably four release cycles, assuming release cycles are 6 weeks.  The
reason is that reviews will be needed and those will take time.

If we wanted to write an in-place migration helper, I'd expect another
two cycles.  Writing one that preserved the existing algorithm and just
added the mapping for the other algorithm would be easier because it
wouldn't require rewriting the `objects` directory.

> If some of us could help you, how could we best help?

I would love someone to start picking up patches from the early part of
the series, rebasing them onto `master`, fixing up any conflicts, and
polishing them, and then sending them in.  The `sha256-interop-part-2`
series would be great for that.

Just let me know if you want to do this and then we won't conflict.

> Could you say which company is interested in helping with this? Would
> that company be willing to work openly with others on this?

I'd rather not disclose that without the permission of the person making
the offer.  I'll just say that a respected contributor and member of the
community offered to have some of the work done on their company's dime
by a person who is also known to the list. They did note that there
would be a delay before starting, so it wouldn't happen right away.

I feel confident that the contributor in question would be willing to
collaborate with others in getting this work done because that's the
kind of person they are and obviously it would be in everyone's interest
to do that.

> Are there some tests or kinds of automated ways to check that things
> work as expected under realistic conditions like:
> 
> - using real world repos (large ones, old ones, with submodules, etc),
> - mixing a number of new and old clients and servers,
> - interacting with other implementations (JGit, libgit2, gitoxide,
> forges, CI, etc)?

What I have done to test this is `git clone --object-format=sha256:sha1
https://github.com/bk2204/lawn.git` and then pushed to a SHA-256
repository.  That's just a personal project of mine that doesn't contain
submodules, but it's what we've used for testing at work and we have
several internal copies of the SHA-256 version of that repo.  It clearly
interoperates with GitHub on a SHA-1-only and SHA-256-only basis, but
there's no support for the interoperability on the server-side yet.
There are also tests for the interoperability code in t1017 which are
reasonably comprehensive.

Once we have in-place migration, I would like to test it with git.git,
since I think that would be a great real-world testcase.
-- 
brian m. carlson (they/them)
Toronto, Ontario, CA

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 325 bytes --]

Re: [RFC PATCH 0/4] sign a SHA-256 digest of the tree in commits and tags

[Junio C Hamano]

"brian m. carlson" <sandals@crustytoothpaste.net> writes:

> The reason more of the interoperability work has not gone upstream is
> because the pluggable ODB work has really ended up breaking a lot of
> things[0], so sending almost anything requires a bunch of rebasing and
> fixing, and I'm presently very burnt out, so I'm doing very little
> coding in my free time and doing more cycling, reading, and Factorio:
> Space Age.

If your time were corporate-funded, and if I declared that we would
accept no changes other than the SHA-256 interoperability work and
perhaps other low-impact changes, and that we would give anyone
helping with the SHA-256 interoperability work the power to veto any
topics that may interfere with quick integration of their work for N
months, would it have worked better, I wonder?

Such an arrangement certainly requires buy-in from other
stakeholders.  Employers who fund scalability work would not only
have to wait their turn, but might also need to be convinced to
divert their resources to help this effort, so that the magic
number N becomes smaller and they get their turn sooner, for
example.

Re: [RFC PATCH 0/4] sign a SHA-256 digest of the tree in commits and tags

[brian m. carlson]

[-- Attachment #1: Type: text/plain, Size: 2328 bytes --]

On 2026-10-06 at 22:38:37, Junio C Hamano wrote:
> If your time were corporate-funded, and if I declared that we would
> accept no changes other than the SHA-256 interoperability work and
> perhaps other low-impact changes, and that we would give anyone
> helping with the SHA-256 interoperability work the power to veto any
> topics that may interfere with quick integration of their work for N
> months, would it have worked better, I wonder?

It might have.  I don't want to say that the ODB work and other
in-flight topics aren't valuable because I feel the opposite, in fact,
but they just make things a moving target and if I'm doing less things
in my personal time, it makes it hard to keep up.

As I mentioned, one of the main impediments to my time being
corporate-funded at the moment is that I can't send out patches from
$DAYJOB because we're forced to use Outlook, which will corrupt patches,
and I don't want to send out work patches from my personal address.  I
don't mind if other people wanted to send out those patches, though, so
that kind of collaboration could work if I could get my employer to
agree (which is likely, given the fact that I previously spent time
working on it, but not guaranteed).

I think if we could get someone to polish and upstream patches while I
work on the next steps at work, that might work well, but of course I
don't want to be very prescriptive about how others contribute.  As I
said, there's plenty of things that need to be done such that we can
have several people working on things and I'm grateful for any
assistance I can get.  Even someone rebasing things, resolving
conflicts, and fixing tests would be super helpful.

One thing is that we would need reviews if we want to get patches merged
in a timely manner and that's kind of difficult at the moment.  That was
an issue for the original SHA-256 work, in fact, as well.

> Such an arrangement certainly requires buy-in from other
> stakeholders.  Employers who fund scalability work would not only
> have to wait their turn, but might also need to be convinced to
> divert their resources to help this effort, so that the magic
> number N becomes smaller and they get their turn sooner, for
> example.

Of course.
-- 
brian m. carlson (they/them)
Toronto, Ontario, CA

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 325 bytes --]

Re: [RFC PATCH 0/4] sign a SHA-256 digest of the tree in commits and tags

[Christian Couder]

On Wed, Oct 7, 2026 at 12:26 AM brian m. carlson
<sandals@crustytoothpaste.net> wrote:
>
> On 2026-10-06 at 09:00:45, Christian Couder wrote:

> > Is the current state of the work publicly available somewhere? Or
> > could you make it publicly available somewhere? (Fine if it's only as
> > patches in a tarball.)
>
> Yeah, it's at https://github.com/bk2204/git.git as `sha256-interop`.

Thanks.

[...]

> > If some of us could help you, how could we best help?
>
> I would love someone to start picking up patches from the early part of
> the series, rebasing them onto `master`, fixing up any conflicts, and
> polishing them, and then sending them in.  The `sha256-interop-part-2`
> series would be great for that.
>
> Just let me know if you want to do this and then we won't conflict.

I am willing to help, but it's not likely I will have a lot of time to
work on it before the end of next month. Anyway let me see if I can
upstream some parts of the `sha256-interop-part-2` series this week or
next week...

Thanks,
Christian.

Re: [RFC PATCH 0/4] sign a SHA-256 digest of the tree in commits and tags

[brian m. carlson]

[-- Attachment #1: Type: text/plain, Size: 559 bytes --]

On 2026-10-07 at 12:26:39, Christian Couder wrote:
> I am willing to help, but it's not likely I will have a lot of time to
> work on it before the end of next month. Anyway let me see if I can
> upstream some parts of the `sha256-interop-part-2` series this week or
> next week...

I appreciate any assistance possible.  Getting that series upstream
unblocks a lot of stuff because then we'll have pack index v3 and object
map support.  That will allow a lot of work to progress in parallel.
-- 
brian m. carlson (they/them)
Toronto, Ontario, CA

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 325 bytes --]