[PATCH 0/4] hash.h: support choosing a separate SHA-1 for non-cryptographic uses

[PATCH 0/4] hash.h: support choosing a separate SHA-1 for non-cryptographic uses

[Taylor Blau]

This series adds a build-time knob to allow selecting an alternative
SHA-1 implementation for non-cryptographic hashing within Git, starting
with the `hashwrite()` family of functions.

This series is the result of starting to roll out verbatim multi-pack
reuse within GitHub's infrastructure. I noticed that on larger
repositories, it is harder thus far to measure a CPU speed-up on clones
where multi-pack reuse is enabled.

After some profiling, I noticed that we spend a significant amount of
time in hashwrite(), which is not all that surprising. But much of that
time is wasted in GitHub's infrastructure, since we are using the same
collision-detecting SHA-1 implementation to produce a trailing checksum
for the pack which does not need to be cryptographically secure.

This series teaches a new set of build-time knobs: OPENSSL_SHA1_FAST,
BLK_SHA1_FAST, and APPLE_COMMON_CRYPTO_SHA1_FAST, which can be used to
select an alterantive SHA-1 implementation for non-cryptographic uses
within Git.

The series is laid out as follows:

  - The first two patches are preparatory, allowing us to include
    multiple SHA-1 wrapper headers and adding scaffolding functions for
    the _fast() variants, respectively.

  - The third patch introduces the build-time knobs for selecting which
    SHA-1 implementation is used for non-cryptographic purposes.

  - The fourth and final patch updates the hashwrite() implementation to
    use the _fast() variants.

From the final commit, a clone of the kernel which took 19.219 seconds
on my machine with no build-time modifications now only takes 11.597
seconds when compiling with OPENSSL_SHA1_FAST=1, for a ~39.7% speed-up.
On a freshly-repacked copy of git.git, the time goes from 758ms to 333ms
for a ~56% speed-up.

Thanks in advance for your review!

Taylor Blau (4):
  sha1: do not redefine `platform_SHA_CTX` and friends
  hash.h: scaffolding for _fast hashing variants
  Makefile: allow specifying a SHA-1 for non-cryptographic uses
  csum-file.c: use fast SHA-1 implementation when available

 Makefile          | 25 ++++++++++++++++++
 block-sha1/sha1.h |  2 ++
 csum-file.c       | 18 ++++++-------
 hash.h            | 67 +++++++++++++++++++++++++++++++++++++++++++++++
 object-file.c     | 42 +++++++++++++++++++++++++++++
 sha1/openssl.h    |  2 ++
 sha1dc_git.h      |  3 +++
 7 files changed, 150 insertions(+), 9 deletions(-)


base-commit: 159f2d50e75c17382c9f4eb7cbda671a6fa612d1
-- 
2.46.0.425.ge8f5cbd280c

[PATCH 1/4] sha1: do not redefine `platform_SHA_CTX` and friends

[Taylor Blau]

Our in-tree SHA-1 wrappers all define platform_SHA_CTX and related
macros to point at the opaque "context" type, init, update, and similar
functions for each specific implementation.

In hash.h, we use these platform_ variables to set up the function
pointers for, e.g., the_hash_algo->init_fn(), etc.

But while these header files have a header-specific macro that prevents
them declaring their structs / functions multiple times, they
unconditionally define the platform variables, making it impossible to
load multiple SHA-1 implementations at once.

As a prerequisite for loading a separate SHA-1 implementation for
non-cryptographic uses, only define the platform_ variables if they have
not already been defined.

Signed-off-by: Taylor Blau <me@ttaylorr.com>
---
 block-sha1/sha1.h | 2 ++
 sha1/openssl.h    | 2 ++
 sha1dc_git.h      | 3 +++
 3 files changed, 7 insertions(+)

diff --git a/block-sha1/sha1.h b/block-sha1/sha1.h
index 9fb0441b988..47bb9166368 100644
--- a/block-sha1/sha1.h
+++ b/block-sha1/sha1.h
@@ -16,7 +16,9 @@ void blk_SHA1_Init(blk_SHA_CTX *ctx);
 void blk_SHA1_Update(blk_SHA_CTX *ctx, const void *dataIn, size_t len);
 void blk_SHA1_Final(unsigned char hashout[20], blk_SHA_CTX *ctx);
 
+#ifndef platform_SHA_CTX
 #define platform_SHA_CTX	blk_SHA_CTX
 #define platform_SHA1_Init	blk_SHA1_Init
 #define platform_SHA1_Update	blk_SHA1_Update
 #define platform_SHA1_Final	blk_SHA1_Final
+#endif
diff --git a/sha1/openssl.h b/sha1/openssl.h
index 006c1f4ba54..1038af47daf 100644
--- a/sha1/openssl.h
+++ b/sha1/openssl.h
@@ -40,10 +40,12 @@ static inline void openssl_SHA1_Clone(struct openssl_SHA1_CTX *dst,
 	EVP_MD_CTX_copy_ex(dst->ectx, src->ectx);
 }
 
+#ifndef platform_SHA_CTX
 #define platform_SHA_CTX openssl_SHA1_CTX
 #define platform_SHA1_Init openssl_SHA1_Init
 #define platform_SHA1_Clone openssl_SHA1_Clone
 #define platform_SHA1_Update openssl_SHA1_Update
 #define platform_SHA1_Final openssl_SHA1_Final
+#endif
 
 #endif /* SHA1_OPENSSL_H */
diff --git a/sha1dc_git.h b/sha1dc_git.h
index 60e3ce84395..f6f880cabea 100644
--- a/sha1dc_git.h
+++ b/sha1dc_git.h
@@ -18,7 +18,10 @@ void git_SHA1DCFinal(unsigned char [20], SHA1_CTX *);
 void git_SHA1DCUpdate(SHA1_CTX *ctx, const void *data, unsigned long len);
 
 #define platform_SHA_IS_SHA1DC /* used by "test-tool sha1-is-sha1dc" */
+
+#ifndef platform_SHA_CTX
 #define platform_SHA_CTX SHA1_CTX
 #define platform_SHA1_Init git_SHA1DCInit
 #define platform_SHA1_Update git_SHA1DCUpdate
 #define platform_SHA1_Final git_SHA1DCFinal
+#endif
-- 
2.46.0.425.ge8f5cbd280c

Re: [PATCH 1/4] sha1: do not redefine `platform_SHA_CTX` and friends

[Patrick Steinhardt]

On Sun, Sep 01, 2024 at 12:03:21PM -0400, Taylor Blau wrote:
> Our in-tree SHA-1 wrappers all define platform_SHA_CTX and related
> macros to point at the opaque "context" type, init, update, and similar
> functions for each specific implementation.
> 
> In hash.h, we use these platform_ variables to set up the function
> pointers for, e.g., the_hash_algo->init_fn(), etc.
> 
> But while these header files have a header-specific macro that prevents
> them declaring their structs / functions multiple times, they
> unconditionally define the platform variables, making it impossible to
> load multiple SHA-1 implementations at once.
> 
> As a prerequisite for loading a separate SHA-1 implementation for
> non-cryptographic uses, only define the platform_ variables if they have
> not already been defined.

So we now pick the first hash we find as platform hash, whereas
previously we would have always picked the last one? Hum, okay. A bit
curious, but let's read on.

Patrick

Re: [PATCH 1/4] sha1: do not redefine `platform_SHA_CTX` and friends

[Taylor Blau]

On Mon, Sep 02, 2024 at 03:41:20PM +0200, Patrick Steinhardt wrote:
> So we now pick the first hash we find as platform hash, whereas
> previously we would have always picked the last one? Hum, okay. A bit
> curious, but let's read on.

In a pre-_FAST SHA-1 environment, we only ever #include one of these to
begin with, so the order doesn't matter. After this series, we may
include two SHA-1 implementations, but we include the non-_FAST one
first, so we'll always pick that one as the default platform
implementation.

Thanks,
Taylor

[PATCH 2/4] hash.h: scaffolding for _fast hashing variants

[Taylor Blau]

Git's default SHA-1 implementation is collision-detecting, which hardens
us against known SHA-1 attacks against Git objects. This makes Git
object writes safer at the expense of some speed when hashing through
the collision-detecting implementation, which is slower than
non-collision detecting alternatives.

Prepare for loading a separate "fast" SHA-1 implementation that can be
used for non-cryptographic purposes, like computing the checksum of
files that use the hashwrite() API.

This commit does not actually introduce any new compile-time knobs to
control which implementation is used as the fast SHA-1 variant, but does
add scaffolding so that the "git_hash_algo" structure has five new
function pointers which are "fast" variants of the five existing
hashing-related function pointers:

  - git_hash_init_fn fast_init_fn
  - git_hash_clone_fn fast_clone_fn
  - git_hash_update_fn fast_update_fn
  - git_hash_final_fn fast_final_fn
  - git_hash_final_oid_fn fast_final_oid_fn

The following commit will introduce compile-time knobs to specify which
SHA-1 implementation is used for non-cryptographic uses.

Signed-off-by: Taylor Blau <me@ttaylorr.com>
---
 hash.h        | 42 ++++++++++++++++++++++++++++++++++++++++++
 object-file.c | 42 ++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 84 insertions(+)

diff --git a/hash.h b/hash.h
index 72ffbc862e5..f255e5c1e8a 100644
--- a/hash.h
+++ b/hash.h
@@ -44,14 +44,32 @@
 #define platform_SHA1_Final    	SHA1_Final
 #endif
 
+#ifndef platform_SHA_CTX_fast
+#define platform_SHA_CTX_fast	  platform_SHA_CTX
+#define platform_SHA1_Init_fast	  platform_SHA1_Init
+#define platform_SHA1_Update_fast platform_SHA1_Update
+#define platform_SHA1_Final_fast  platform_SHA1_Final
+#ifdef platform_SHA1_Clone
+#define platform_SHA1_Clone_fast  platform_SHA1_Clone
+#endif
+#endif
+
 #define git_SHA_CTX		platform_SHA_CTX
 #define git_SHA1_Init		platform_SHA1_Init
 #define git_SHA1_Update		platform_SHA1_Update
 #define git_SHA1_Final		platform_SHA1_Final
 
+#define git_SHA_CTX_fast	platform_SHA_CTX_fast
+#define git_SHA1_Init_fast	platform_SHA1_Init_fast
+#define git_SHA1_Update_fast	platform_SHA1_Update_fast
+#define git_SHA1_Final_fast	platform_SHA1_Final_fast
+
 #ifdef platform_SHA1_Clone
 #define git_SHA1_Clone	platform_SHA1_Clone
 #endif
+#ifdef platform_SHA1_Clone_fast
+#define git_SHA1_Clone_fast	platform_SHA1_Clone_fast
+#endif
 
 #ifndef platform_SHA256_CTX
 #define platform_SHA256_CTX	SHA256_CTX
@@ -81,6 +99,13 @@ static inline void git_SHA1_Clone(git_SHA_CTX *dst, const git_SHA_CTX *src)
 	memcpy(dst, src, sizeof(*dst));
 }
 #endif
+#ifndef SHA1_NEEDS_CLONE_HELPER_FAST
+static inline void git_SHA1_Clone_fast(git_SHA_CTX_fast *dst,
+				       const git_SHA_CTX_fast *src)
+{
+	memcpy(dst, src, sizeof(*dst));
+}
+#endif
 
 #ifndef SHA256_NEEDS_CLONE_HELPER
 static inline void git_SHA256_Clone(git_SHA256_CTX *dst, const git_SHA256_CTX *src)
@@ -178,6 +203,8 @@ enum get_oid_result {
 /* A suitably aligned type for stack allocations of hash contexts. */
 union git_hash_ctx {
 	git_SHA_CTX sha1;
+	git_SHA_CTX_fast sha1_fast;
+
 	git_SHA256_CTX sha256;
 };
 typedef union git_hash_ctx git_hash_ctx;
@@ -222,6 +249,21 @@ struct git_hash_algo {
 	/* The hash finalization function for object IDs. */
 	git_hash_final_oid_fn final_oid_fn;
 
+	/* The fast hash initialization function. */
+	git_hash_init_fn fast_init_fn;
+
+	/* The fast hash context cloning function. */
+	git_hash_clone_fn fast_clone_fn;
+
+	/* The fast hash update function. */
+	git_hash_update_fn fast_update_fn;
+
+	/* The fast hash finalization function. */
+	git_hash_final_fn fast_final_fn;
+
+	/* The fast hash finalization function for object IDs. */
+	git_hash_final_oid_fn fast_final_oid_fn;
+
 	/* The OID of the empty tree. */
 	const struct object_id *empty_tree;
 
diff --git a/object-file.c b/object-file.c
index c5994202ba0..9691292ef5a 100644
--- a/object-file.c
+++ b/object-file.c
@@ -115,6 +115,33 @@ static void git_hash_sha1_final_oid(struct object_id *oid, git_hash_ctx *ctx)
 	oid->algo = GIT_HASH_SHA1;
 }
 
+static void git_hash_sha1_init_fast(git_hash_ctx *ctx)
+{
+	git_SHA1_Init_fast(&ctx->sha1_fast);
+}
+
+static void git_hash_sha1_clone_fast(git_hash_ctx *dst, const git_hash_ctx *src)
+{
+	git_SHA1_Clone_fast(&dst->sha1_fast, &src->sha1_fast);
+}
+
+static void git_hash_sha1_update_fast(git_hash_ctx *ctx, const void *data,
+				      size_t len)
+{
+	git_SHA1_Update_fast(&ctx->sha1_fast, data, len);
+}
+
+static void git_hash_sha1_final_fast(unsigned char *hash, git_hash_ctx *ctx)
+{
+	git_SHA1_Final_fast(hash, &ctx->sha1_fast);
+}
+
+static void git_hash_sha1_final_oid_fast(struct object_id *oid, git_hash_ctx *ctx)
+{
+	git_SHA1_Final_fast(oid->hash, &ctx->sha1_fast);
+	memset(oid->hash + GIT_SHA1_RAWSZ, 0, GIT_MAX_RAWSZ - GIT_SHA1_RAWSZ);
+	oid->algo = GIT_HASH_SHA1;
+}
 
 static void git_hash_sha256_init(git_hash_ctx *ctx)
 {
@@ -189,6 +216,11 @@ const struct git_hash_algo hash_algos[GIT_HASH_NALGOS] = {
 		.update_fn = git_hash_unknown_update,
 		.final_fn = git_hash_unknown_final,
 		.final_oid_fn = git_hash_unknown_final_oid,
+		.fast_init_fn = git_hash_unknown_init,
+		.fast_clone_fn = git_hash_unknown_clone,
+		.fast_update_fn = git_hash_unknown_update,
+		.fast_final_fn = git_hash_unknown_final,
+		.fast_final_oid_fn = git_hash_unknown_final_oid,
 		.empty_tree = NULL,
 		.empty_blob = NULL,
 		.null_oid = NULL,
@@ -204,6 +236,11 @@ const struct git_hash_algo hash_algos[GIT_HASH_NALGOS] = {
 		.update_fn = git_hash_sha1_update,
 		.final_fn = git_hash_sha1_final,
 		.final_oid_fn = git_hash_sha1_final_oid,
+		.fast_init_fn = git_hash_sha1_init_fast,
+		.fast_clone_fn = git_hash_sha1_clone_fast,
+		.fast_update_fn = git_hash_sha1_update_fast,
+		.fast_final_fn = git_hash_sha1_final_fast,
+		.fast_final_oid_fn = git_hash_sha1_final_oid_fast,
 		.empty_tree = &empty_tree_oid,
 		.empty_blob = &empty_blob_oid,
 		.null_oid = &null_oid_sha1,
@@ -219,6 +256,11 @@ const struct git_hash_algo hash_algos[GIT_HASH_NALGOS] = {
 		.update_fn = git_hash_sha256_update,
 		.final_fn = git_hash_sha256_final,
 		.final_oid_fn = git_hash_sha256_final_oid,
+		.fast_init_fn = git_hash_sha256_init,
+		.fast_clone_fn = git_hash_sha256_clone,
+		.fast_update_fn = git_hash_sha256_update,
+		.fast_final_fn = git_hash_sha256_final,
+		.fast_final_oid_fn = git_hash_sha256_final_oid,
 		.empty_tree = &empty_tree_oid_sha256,
 		.empty_blob = &empty_blob_oid_sha256,
 		.null_oid = &null_oid_sha256,
-- 
2.46.0.425.ge8f5cbd280c

Re: [PATCH 2/4] hash.h: scaffolding for _fast hashing variants

[Patrick Steinhardt]

On Sun, Sep 01, 2024 at 12:03:24PM -0400, Taylor Blau wrote:
> Git's default SHA-1 implementation is collision-detecting, which hardens
> us against known SHA-1 attacks against Git objects. This makes Git
> object writes safer at the expense of some speed when hashing through
> the collision-detecting implementation, which is slower than
> non-collision detecting alternatives.
> 
> Prepare for loading a separate "fast" SHA-1 implementation that can be
> used for non-cryptographic purposes, like computing the checksum of
> files that use the hashwrite() API.
> 
> This commit does not actually introduce any new compile-time knobs to
> control which implementation is used as the fast SHA-1 variant, but does
> add scaffolding so that the "git_hash_algo" structure has five new
> function pointers which are "fast" variants of the five existing
> hashing-related function pointers:
> 
>   - git_hash_init_fn fast_init_fn
>   - git_hash_clone_fn fast_clone_fn
>   - git_hash_update_fn fast_update_fn
>   - git_hash_final_fn fast_final_fn
>   - git_hash_final_oid_fn fast_final_oid_fn
> 
> The following commit will introduce compile-time knobs to specify which
> SHA-1 implementation is used for non-cryptographic uses.

While the property we care about in the context of this patch series
indeed is that the second hash is faster, I think the more important
property is that it's insecure. If I were seeing two APIs, one labelled
fast and one labelled slow, I would of course pick the fast one. So I
wonder whether we should rename things accordingly so that developers
aren't intrigued to pick the fast one without thinking, and also to have
a more useful signal that stands out to reviewers.

> Signed-off-by: Taylor Blau <me@ttaylorr.com>
> ---
>  hash.h        | 42 ++++++++++++++++++++++++++++++++++++++++++
>  object-file.c | 42 ++++++++++++++++++++++++++++++++++++++++++
>  2 files changed, 84 insertions(+)
> 
> diff --git a/hash.h b/hash.h
> index 72ffbc862e5..f255e5c1e8a 100644
> --- a/hash.h
> +++ b/hash.h
> @@ -44,14 +44,32 @@
>  #define platform_SHA1_Final    	SHA1_Final
>  #endif
>  
> +#ifndef platform_SHA_CTX_fast
> +#define platform_SHA_CTX_fast	  platform_SHA_CTX
> +#define platform_SHA1_Init_fast	  platform_SHA1_Init
> +#define platform_SHA1_Update_fast platform_SHA1_Update
> +#define platform_SHA1_Final_fast  platform_SHA1_Final
> +#ifdef platform_SHA1_Clone
> +#define platform_SHA1_Clone_fast  platform_SHA1_Clone
> +#endif
> +#endif

We may want to apply our new coding guidelines around nested
preprocessor directives, which should also use indenting.

> @@ -222,6 +249,21 @@ struct git_hash_algo {
>  	/* The hash finalization function for object IDs. */
>  	git_hash_final_oid_fn final_oid_fn;
>  
> +	/* The fast hash initialization function. */

Providing some context here why there are two sets of functions would
help future readers.

> @@ -219,6 +256,11 @@ const struct git_hash_algo hash_algos[GIT_HASH_NALGOS] = {
>  		.update_fn = git_hash_sha256_update,
>  		.final_fn = git_hash_sha256_final,
>  		.final_oid_fn = git_hash_sha256_final_oid,
> +		.fast_init_fn = git_hash_sha256_init,
> +		.fast_clone_fn = git_hash_sha256_clone,
> +		.fast_update_fn = git_hash_sha256_update,
> +		.fast_final_fn = git_hash_sha256_final,
> +		.fast_final_oid_fn = git_hash_sha256_final_oid,
>  		.empty_tree = &empty_tree_oid_sha256,
>  		.empty_blob = &empty_blob_oid_sha256,
>  		.null_oid = &null_oid_sha256,

I was briefly wondering whether we'd rather want to have automatic
fallbacks to the secure alternative when the fast one isn't set. I guess
in the end that's not really worth it though, as it saves us one branch
when not having such a fallback. And we only have so many hashes, so
it's not like this would be a huge pain to maintain.

Patrick

Re: [PATCH 2/4] hash.h: scaffolding for _fast hashing variants

[Junio C Hamano]

Patrick Steinhardt <ps@pks.im> writes:

> While the property we care about in the context of this patch series
> indeed is that the second hash is faster, I think the more important
> property is that it's insecure. If I were seeing two APIs, one labelled
> fast and one labelled slow, I would of course pick the fast one. So I
> wonder whether we should rename things accordingly so that developers
> aren't intrigued to pick the fast one without thinking, and also to have
> a more useful signal that stands out to reviewers.

I do not think this topic is going in the direction it set out to,
but if we are to resurrect it by 

 (1) first to ensure that we won't overwrite existing on-disk files
     and other things as needed to safely swap the tail sum to a
     cryptographically insecure hash function;

 (2) devise a transition plan to use a hash function that computes a
     value that is different from SHA-1 (or SHA-256 for that
     matter); and

 (3) pick a hash function that computes a lot faster but is insecure
     and transition to it.

we will need to clearly label the two hash functions as such.

We may also need to consider similar points if we need to name
pseudo random numbers we use, to clarify the requirement of the
caller (e.g., can a caller that wants security use it?).

Thanks.

Re: [PATCH 2/4] hash.h: scaffolding for _fast hashing variants

[Taylor Blau]

On Tue, Sep 03, 2024 at 10:27:39AM -0700, Junio C Hamano wrote:
> Patrick Steinhardt <ps@pks.im> writes:
>
> > While the property we care about in the context of this patch series
> > indeed is that the second hash is faster, I think the more important
> > property is that it's insecure. If I were seeing two APIs, one labelled
> > fast and one labelled slow, I would of course pick the fast one. So I
> > wonder whether we should rename things accordingly so that developers
> > aren't intrigued to pick the fast one without thinking, and also to have
> > a more useful signal that stands out to reviewers.
>
> I do not think this topic is going in the direction it set out to,
> but if we are to resurrect it by
>
>  (1) first to ensure that we won't overwrite existing on-disk files
>      and other things as needed to safely swap the tail sum to a
>      cryptographically insecure hash function;

I discussed this with brian in the sub-thread where I am talking to
them, but I think this is already the case. The pack is read in
index-pack and the checksum is verified without using the _fast hash
functions, so we would detect:

  - either half of a colliding pair of objects, when reading individual
    objects' contents to determine their SHA-1s, or

  - a colliding pack checksum, when computing the whole pack's checksum
    (which also does not use the _fast variants of these functions), and

  - a mismatched pack checksum, when verifying the pack's checksum
    against the one stored in the pack.

>  (2) devise a transition plan to use a hash function that computes a
>      value that is different from SHA-1 (or SHA-256 for that
>      matter); and
>
>  (3) pick a hash function that computes a lot faster but is insecure
>      and transition to it.

So I do not think that either of these two steps are necessary.

Thanks,
Taylor

Re: [PATCH 2/4] hash.h: scaffolding for _fast hashing variants

[Junio C Hamano]

Taylor Blau <me@ttaylorr.com> writes:

> I discussed this with brian in the sub-thread where I am talking to
> them, but I think this is already the case. The pack is read in
> index-pack and the checksum is verified without using the _fast hash
> functions, so we would detect:
>
>   - either half of a colliding pair of objects, when reading individual
>     objects' contents to determine their SHA-1s, or
>
>   - a colliding pack checksum, when computing the whole pack's checksum
>     (which also does not use the _fast variants of these functions), and
>
>   - a mismatched pack checksum, when verifying the pack's checksum
>     against the one stored in the pack.
>
>>  (2) devise a transition plan to use a hash function that computes a
>>      value that is different from SHA-1 (or SHA-256 for that
>>      matter); and
>>
>>  (3) pick a hash function that computes a lot faster but is insecure
>>      and transition to it.
>
> So I do not think that either of these two steps are necessary.

I suspect that it is a wrong conclusion, as I meant (1) to be
prerequisite for doing (2) and (3), that gives us the real benefit
of being able to go faster than SHA1DC or even SHA-256.  If (1) is
unnecessary (because it is already covered), that is great---we can
directly jump to (2) and (3).

Re: [PATCH 2/4] hash.h: scaffolding for _fast hashing variants

[Taylor Blau]

On Tue, Sep 03, 2024 at 01:47:09PM -0700, Junio C Hamano wrote:
> > So I do not think that either of these two steps are necessary.
>
> I suspect that it is a wrong conclusion, as I meant (1) to be
> prerequisite for doing (2) and (3), that gives us the real benefit
> of being able to go faster than SHA1DC or even SHA-256.  If (1) is
> unnecessary (because it is already covered), that is great---we can
> directly jump to (2) and (3).

Ah, yes, after re-reading your message I am definitely mistaken here. I
think that in the future doing this would be more than worthwhile.

Thanks,
Taylor

Re: [PATCH 2/4] hash.h: scaffolding for _fast hashing variants

[Patrick Steinhardt]

On Tue, Sep 03, 2024 at 01:47:09PM -0700, Junio C Hamano wrote:
> Taylor Blau <me@ttaylorr.com> writes:
> 
> > I discussed this with brian in the sub-thread where I am talking to
> > them, but I think this is already the case. The pack is read in
> > index-pack and the checksum is verified without using the _fast hash
> > functions, so we would detect:
> >
> >   - either half of a colliding pair of objects, when reading individual
> >     objects' contents to determine their SHA-1s, or
> >
> >   - a colliding pack checksum, when computing the whole pack's checksum
> >     (which also does not use the _fast variants of these functions), and
> >
> >   - a mismatched pack checksum, when verifying the pack's checksum
> >     against the one stored in the pack.
> >
> >>  (2) devise a transition plan to use a hash function that computes a
> >>      value that is different from SHA-1 (or SHA-256 for that
> >>      matter); and
> >>
> >>  (3) pick a hash function that computes a lot faster but is insecure
> >>      and transition to it.
> >
> > So I do not think that either of these two steps are necessary.
> 
> I suspect that it is a wrong conclusion, as I meant (1) to be
> prerequisite for doing (2) and (3), that gives us the real benefit
> of being able to go faster than SHA1DC or even SHA-256.  If (1) is
> unnecessary (because it is already covered), that is great---we can
> directly jump to (2) and (3).

Ah, so the idea would be to not introduce SHA1_fast, but instead use a
hash function that is explicitly designed for fast hashing like xxHash
[1]? When you compare numbers I definitely think that this makes quite
some sense as XXH3 for example hashes at 31.5GB/s whereas SHA1 hashes at
0.8GB/s (if you believe the numbers on their site).

Doing this for data structures structur is almost a no-brainer if you
ask me. For packfiles it's a bit more complicated as we also have to
consider backwards compatibility -- a server of course cannot just start
to send packfiles that use xxHash.

Patrick

[1]: https://github.com/Cyan4973/xxHash

Re: [PATCH 2/4] hash.h: scaffolding for _fast hashing variants

[Junio C Hamano]

Patrick Steinhardt <ps@pks.im> writes:

> On Tue, Sep 03, 2024 at 01:47:09PM -0700, Junio C Hamano wrote:
>> ...
>> >>  (2) devise a transition plan to use a hash function that computes a
>> >>      value that is different from SHA-1 (or SHA-256 for that
>> >>      matter); and
>> >>
>> >>  (3) pick a hash function that computes a lot faster but is insecure
>> >>      and transition to it.
>> >
>> > So I do not think that either of these two steps are necessary.
>> 
>> I suspect that it is a wrong conclusion, as I meant (1) to be
>> prerequisite for doing (2) and (3), that gives us the real benefit
>> of being able to go faster than SHA1DC or even SHA-256.  If (1) is
>> unnecessary (because it is already covered), that is great---we can
>> directly jump to (2) and (3).
>
> Ah, so the idea would be to not introduce SHA1_fast, but instead use a
> hash function that is explicitly designed for fast hashing like xxHash
> [1]? When you compare numbers I definitely think that this makes quite
> some sense as XXH3 for example hashes at 31.5GB/s whereas SHA1 hashes at
> 0.8GB/s (if you believe the numbers on their site).

> Doing this for data structures structur is almost a no-brainer if you
> ask me. For packfiles it's a bit more complicated as we also have to
> consider backwards compatibility -- a server of course cannot just start
> to send packfiles that use xxHash.

Yup, that is where the step (2) above comes in.  In the thread,
xxhash was indeed brought up as a viable candidate for the tail sum
for strictly local files.

Re: [PATCH 2/4] hash.h: scaffolding for _fast hashing variants

[Taylor Blau]

On Mon, Sep 02, 2024 at 03:41:24PM +0200, Patrick Steinhardt wrote:
> > This commit does not actually introduce any new compile-time knobs to
> > control which implementation is used as the fast SHA-1 variant, but does
> > add scaffolding so that the "git_hash_algo" structure has five new
> > function pointers which are "fast" variants of the five existing
> > hashing-related function pointers:
> >
> >   - git_hash_init_fn fast_init_fn
> >   - git_hash_clone_fn fast_clone_fn
> >   - git_hash_update_fn fast_update_fn
> >   - git_hash_final_fn fast_final_fn
> >   - git_hash_final_oid_fn fast_final_oid_fn
> >
> > The following commit will introduce compile-time knobs to specify which
> > SHA-1 implementation is used for non-cryptographic uses.
>
> While the property we care about in the context of this patch series
> indeed is that the second hash is faster, I think the more important
> property is that it's insecure. If I were seeing two APIs, one labelled
> fast and one labelled slow, I would of course pick the fast one. So I
> wonder whether we should rename things accordingly so that developers
> aren't intrigued to pick the fast one without thinking, and also to have
> a more useful signal that stands out to reviewers.

I tried to come up with a different name myself when writing this
series, and wasn't happy with any of the others that I came up with. I
thought of "insecure_init_fn()", or "non_cryptographic_init_fn()". The
first one appears scarier than the second, but both are mouthfuls.

As a middle-ground, I updated the comments to say "fast /
non-cryptographic" in places where they just said "fast" previously. Let
me know if you think that's sufficient, otherwise I can try and come up
with some more names.

> We may want to apply our new coding guidelines around nested
> preprocessor directives, which should also use indenting.

Gotcha.

> > @@ -222,6 +249,21 @@ struct git_hash_algo {
> >  	/* The hash finalization function for object IDs. */
> >  	git_hash_final_oid_fn final_oid_fn;
> >
> > +	/* The fast hash initialization function. */
>
> Providing some context here why there are two sets of functions would
> help future readers.

Very fair, will adjust (as above).

Thanks,
Taylor

[PATCH 3/4] Makefile: allow specifying a SHA-1 for non-cryptographic uses

[Taylor Blau]

Introduce _FAST variants of the OPENSSL_SHA1, BLK_SHA1, and
APPLE_COMMON_CRYPTO_SHA1 compile-time knobs which indicate which SHA-1
implementation is to be used for non-cryptographic uses.

There are a couple of small implementation notes worth mentioning:

  - There is no way to select the collision detecting SHA-1 as the
    "fast" fallback, since the fast fallback is only for
    non-cryptographic uses, and is meant to be faster than our
    collision-detecting implementation.

  - There are no similar knobs for SHA-256, since no collision attacks
    are presently known and thus no collision-detecting implementations
    actually exist.

Signed-off-by: Taylor Blau <me@ttaylorr.com>
---
 Makefile | 25 +++++++++++++++++++++++++
 hash.h   | 25 +++++++++++++++++++++++++
 2 files changed, 50 insertions(+)

diff --git a/Makefile b/Makefile
index e298c8b55ec..d24f9088802 100644
--- a/Makefile
+++ b/Makefile
@@ -517,6 +517,10 @@ include shared.mak
 # Define APPLE_COMMON_CRYPTO_SHA1 to use Apple's CommonCrypto for
 # SHA-1.
 #
+# Define the same Makefile knobs as above, but suffixed with _FAST to
+# use the corresponding implementations for "fast" SHA-1 hashing for
+# non-cryptographic purposes.
+#
 # If don't enable any of the *_SHA1 settings in this section, Git will
 # default to its built-in sha1collisiondetection library, which is a
 # collision-detecting sha1 This is slower, but may detect attempted
@@ -1982,6 +1986,27 @@ endif
 endif
 endif
 
+ifdef OPENSSL_SHA1_FAST
+ifndef OPENSSL_SHA1
+	EXTLIBS += $(LIB_4_CRYPTO)
+	BASIC_CFLAGS += -DSHA1_OPENSSL_FAST
+endif
+else
+ifdef BLK_SHA1_FAST
+ifndef BLK_SHA1
+	LIB_OBJS += block-sha1/sha1.o
+	BASIC_CFLAGS += -DSHA1_BLK_FAST
+endif
+else
+ifdef APPLE_COMMON_CRYPTO_SHA1_FAST
+ifndef APPLE_COMMON_CRYPTO_SHA1
+	COMPAT_CFLAGS += -DCOMMON_DIGEST_FOR_OPENSSL
+	BASIC_CFLAGS += -DSHA1_APPLE_FAST
+endif
+endif
+endif
+endif
+
 ifdef OPENSSL_SHA256
 	EXTLIBS += $(LIB_4_CRYPTO)
 	BASIC_CFLAGS += -DSHA256_OPENSSL
diff --git a/hash.h b/hash.h
index f255e5c1e8a..450e579b405 100644
--- a/hash.h
+++ b/hash.h
@@ -15,6 +15,31 @@
 #include "block-sha1/sha1.h"
 #endif
 
+#if defined(SHA1_APPLE_FAST)
+#include <CommonCrypto/CommonDigest.h>
+#define platform_SHA_CTX_fast CC_SHA1_CTX
+#define platform_SHA1_Init_fast CC_SHA1_Init
+#define platform_SHA1_Update_fast CC_SHA1_Update
+#define platform_SHA1_Final_fast CC_SHA1_Final
+#elif defined(SHA1_OPENSSL_FAST)
+#  include <openssl/sha.h>
+#  if defined(OPENSSL_API_LEVEL) && OPENSSL_API_LEVEL >= 3
+#    define SHA1_NEEDS_CLONE_HELPER_FAST
+#    include "sha1/openssl.h"
+#  endif
+#  define platform_SHA_CTX_fast openssl_SHA1_CTX
+#  define platform_SHA1_Init_fast openssl_SHA1_Init
+#  define platform_SHA1_Clone_fast openssl_SHA1_Clone
+#  define platform_SHA1_Update_fast openssl_SHA1_Update
+#  define platform_SHA1_Final_fast openssl_SHA1_Final
+#elif defined(SHA1_BLK_FAST)
+#include "block-sha1/sha1.h"
+#define platform_SHA_CTX_fast blk_SHA_CTX
+#define platform_SHA1_Init_fast blk_SHA1_Init
+#define platform_SHA1_Update_fast blk_SHA1_Update
+#define platform_SHA1_Final_fast blk_SHA1_Final
+#endif
+
 #if defined(SHA256_NETTLE)
 #include "sha256/nettle.h"
 #elif defined(SHA256_GCRYPT)
-- 
2.46.0.425.ge8f5cbd280c

Re: [PATCH 3/4] Makefile: allow specifying a SHA-1 for non-cryptographic uses

[Patrick Steinhardt]

On Sun, Sep 01, 2024 at 12:03:28PM -0400, Taylor Blau wrote:
> diff --git a/Makefile b/Makefile
> index e298c8b55ec..d24f9088802 100644
> @@ -1982,6 +1986,27 @@ endif
>  endif
>  endif
>  
> +ifdef OPENSSL_SHA1_FAST
> +ifndef OPENSSL_SHA1
> +	EXTLIBS += $(LIB_4_CRYPTO)
> +	BASIC_CFLAGS += -DSHA1_OPENSSL_FAST
> +endif
> +else
> +ifdef BLK_SHA1_FAST
> +ifndef BLK_SHA1
> +	LIB_OBJS += block-sha1/sha1.o
> +	BASIC_CFLAGS += -DSHA1_BLK_FAST
> +endif
> +else
> +ifdef APPLE_COMMON_CRYPTO_SHA1_FAST
> +ifndef APPLE_COMMON_CRYPTO_SHA1
> +	COMPAT_CFLAGS += -DCOMMON_DIGEST_FOR_OPENSSL
> +	BASIC_CFLAGS += -DSHA1_APPLE_FAST
> +endif
> +endif
> +endif
> +endif
> +

What a cascade of `endif`s :)

Do we also want to wire up support in config.mak.uname such that the
fast variants are default-enabled? Or is there a good reason to not do
that?

> diff --git a/hash.h b/hash.h
> index f255e5c1e8a..450e579b405 100644
> --- a/hash.h
> +++ b/hash.h
> @@ -15,6 +15,31 @@
>  #include "block-sha1/sha1.h"
>  #endif
>  
> +#if defined(SHA1_APPLE_FAST)
> +#include <CommonCrypto/CommonDigest.h>
> +#define platform_SHA_CTX_fast CC_SHA1_CTX
> +#define platform_SHA1_Init_fast CC_SHA1_Init
> +#define platform_SHA1_Update_fast CC_SHA1_Update
> +#define platform_SHA1_Final_fast CC_SHA1_Final
> +#elif defined(SHA1_OPENSSL_FAST)
> +#  include <openssl/sha.h>
> +#  if defined(OPENSSL_API_LEVEL) && OPENSSL_API_LEVEL >= 3
> +#    define SHA1_NEEDS_CLONE_HELPER_FAST
> +#    include "sha1/openssl.h"
> +#  endif
> +#  define platform_SHA_CTX_fast openssl_SHA1_CTX
> +#  define platform_SHA1_Init_fast openssl_SHA1_Init
> +#  define platform_SHA1_Clone_fast openssl_SHA1_Clone
> +#  define platform_SHA1_Update_fast openssl_SHA1_Update
> +#  define platform_SHA1_Final_fast openssl_SHA1_Final
> +#elif defined(SHA1_BLK_FAST)
> +#include "block-sha1/sha1.h"
> +#define platform_SHA_CTX_fast blk_SHA_CTX
> +#define platform_SHA1_Init_fast blk_SHA1_Init
> +#define platform_SHA1_Update_fast blk_SHA1_Update
> +#define platform_SHA1_Final_fast blk_SHA1_Final
> +#endif
> +
>  #if defined(SHA256_NETTLE)
>  #include "sha256/nettle.h"
>  #elif defined(SHA256_GCRYPT)

Curiously, some of the nested statements here are indented whereas
others aren't. We should aim to make that consistent.

Patrick

Re: [PATCH 3/4] Makefile: allow specifying a SHA-1 for non-cryptographic uses

[Taylor Blau]

On Mon, Sep 02, 2024 at 03:41:28PM +0200, Patrick Steinhardt wrote:
> > +endif
> > +endif
> > +endif
> > +endif
> > +
>
> What a cascade of `endif`s :)

Heh, indeed. These are copy/pasted from the hunk below this one, so
nothing new here.

> Do we also want to wire up support in config.mak.uname such that the
> fast variants are default-enabled? Or is there a good reason to not do
> that?

I thought that I might consider doing that in a separate series, if at
all. I would like have users opt-in to the new behavior rather than
imposing any change on them in this series.

> > diff --git a/hash.h b/hash.h
> > index f255e5c1e8a..450e579b405 100644
> > --- a/hash.h
> > +++ b/hash.h
> > @@ -15,6 +15,31 @@
> >  #include "block-sha1/sha1.h"
> >  #endif
> >
> > +#if defined(SHA1_APPLE_FAST)
> > +#include <CommonCrypto/CommonDigest.h>
> > +#define platform_SHA_CTX_fast CC_SHA1_CTX
> > +#define platform_SHA1_Init_fast CC_SHA1_Init
> > +#define platform_SHA1_Update_fast CC_SHA1_Update
> > +#define platform_SHA1_Final_fast CC_SHA1_Final
> > +#elif defined(SHA1_OPENSSL_FAST)
> > +#  include <openssl/sha.h>
> > +#  if defined(OPENSSL_API_LEVEL) && OPENSSL_API_LEVEL >= 3
> > +#    define SHA1_NEEDS_CLONE_HELPER_FAST
> > +#    include "sha1/openssl.h"
> > +#  endif
> > +#  define platform_SHA_CTX_fast openssl_SHA1_CTX
> > +#  define platform_SHA1_Init_fast openssl_SHA1_Init
> > +#  define platform_SHA1_Clone_fast openssl_SHA1_Clone
> > +#  define platform_SHA1_Update_fast openssl_SHA1_Update
> > +#  define platform_SHA1_Final_fast openssl_SHA1_Final
> > +#elif defined(SHA1_BLK_FAST)
> > +#include "block-sha1/sha1.h"
> > +#define platform_SHA_CTX_fast blk_SHA_CTX
> > +#define platform_SHA1_Init_fast blk_SHA1_Init
> > +#define platform_SHA1_Update_fast blk_SHA1_Update
> > +#define platform_SHA1_Final_fast blk_SHA1_Final
> > +#endif
> > +
> >  #if defined(SHA256_NETTLE)
> >  #include "sha256/nettle.h"
> >  #elif defined(SHA256_GCRYPT)
>
> Curiously, some of the nested statements here are indented whereas
> others aren't. We should aim to make that consistent.

Sure, this one was also copy/pasted from the block above, but I'll
adjust the new one accordingly.

Thanks,
Taylor

[PATCH 4/4] csum-file.c: use fast SHA-1 implementation when available

[Taylor Blau]

Update hashwrite() and friends to use the fast_-variants of hashing
functions, calling for e.g., "the_hash_algo->fast_update_fn()" instead
of "the_hash_algo->update_fn()".

These callers only use the_hash_algo to produce a checksum, which we
depend on for data integrity, but not for cryptographic purposes, so
these callers are safe to use the fast (and potentially non-collision
detecting) SHA-1 implementation.

To time this, I took a freshly packed copy of linux.git, and ran the
following with and without the OPENSSL_SHA1_FAST=1 build-knob. Both
versions were compiled with -O3:

    $ git for-each-ref --format='%(objectname)' refs/heads refs/tags >in
    $ valgrind --tool=callgrind ~/src/git/git-pack-objects \
        --revs --stdout --all-progress --use-bitmap-index <in >/dev/null

Without OPENSSL_SHA1_FAST=1 (that is, using the collision-detecting
SHA-1 implementation for both cryptographic and non-cryptographic
purposes), we spend a significant amount of our instruction count in
hashwrite():

    $ callgrind_annotate --inclusive=yes | grep hashwrite | head -n1
    159,998,868,413 (79.42%)  /home/ttaylorr/src/git/csum-file.c:hashwrite [/home/ttaylorr/src/git/git-pack-objects]

, and the resulting "clone" takes 19.219 seconds of wall clock time,
18.94 seconds of user time and 0.28 seconds of system time.

Compiling with OPENSSL_SHA1_FAST=1, we spend ~60% fewer instructions in
hashwrite():

    $ callgrind_annotate --inclusive=yes | grep hashwrite | head -n1
     59,164,001,176 (58.79%)  /home/ttaylorr/src/git/csum-file.c:hashwrite [/home/ttaylorr/src/git/git-pack-objects]

, and generate the resulting "clone" much faster, in only 11.597 seconds
of wall time, 11.37 seconds of user time, and 0.23 seconds of system
time, for a ~40% speed-up.

Signed-off-by: Taylor Blau <me@ttaylorr.com>
---
 csum-file.c | 18 +++++++++---------
 1 file changed, 9 insertions(+), 9 deletions(-)

diff --git a/csum-file.c b/csum-file.c
index bf82ad8f9f5..cb8c39ecf3a 100644
--- a/csum-file.c
+++ b/csum-file.c
@@ -50,7 +50,7 @@ void hashflush(struct hashfile *f)
 
 	if (offset) {
 		if (!f->skip_hash)
-			the_hash_algo->update_fn(&f->ctx, f->buffer, offset);
+			the_hash_algo->fast_update_fn(&f->ctx, f->buffer, offset);
 		flush(f, f->buffer, offset);
 		f->offset = 0;
 	}
@@ -73,7 +73,7 @@ int finalize_hashfile(struct hashfile *f, unsigned char *result,
 	if (f->skip_hash)
 		hashclr(f->buffer, the_repository->hash_algo);
 	else
-		the_hash_algo->final_fn(f->buffer, &f->ctx);
+		the_hash_algo->fast_final_fn(f->buffer, &f->ctx);
 
 	if (result)
 		hashcpy(result, f->buffer, the_repository->hash_algo);
@@ -128,7 +128,7 @@ void hashwrite(struct hashfile *f, const void *buf, unsigned int count)
 			 * f->offset is necessarily zero.
 			 */
 			if (!f->skip_hash)
-				the_hash_algo->update_fn(&f->ctx, buf, nr);
+				the_hash_algo->fast_update_fn(&f->ctx, buf, nr);
 			flush(f, buf, nr);
 		} else {
 			/*
@@ -174,7 +174,7 @@ static struct hashfile *hashfd_internal(int fd, const char *name,
 	f->name = name;
 	f->do_crc = 0;
 	f->skip_hash = 0;
-	the_hash_algo->init_fn(&f->ctx);
+	the_hash_algo->fast_init_fn(&f->ctx);
 
 	f->buffer_len = buffer_len;
 	f->buffer = xmalloc(buffer_len);
@@ -208,7 +208,7 @@ void hashfile_checkpoint(struct hashfile *f, struct hashfile_checkpoint *checkpo
 {
 	hashflush(f);
 	checkpoint->offset = f->total;
-	the_hash_algo->clone_fn(&checkpoint->ctx, &f->ctx);
+	the_hash_algo->fast_clone_fn(&checkpoint->ctx, &f->ctx);
 }
 
 int hashfile_truncate(struct hashfile *f, struct hashfile_checkpoint *checkpoint)
@@ -219,7 +219,7 @@ int hashfile_truncate(struct hashfile *f, struct hashfile_checkpoint *checkpoint
 	    lseek(f->fd, offset, SEEK_SET) != offset)
 		return -1;
 	f->total = offset;
-	the_hash_algo->clone_fn(&f->ctx, &checkpoint->ctx);
+	the_hash_algo->fast_clone_fn(&f->ctx, &checkpoint->ctx);
 	f->offset = 0; /* hashflush() was called in checkpoint */
 	return 0;
 }
@@ -245,9 +245,9 @@ int hashfile_checksum_valid(const unsigned char *data, size_t total_len)
 	if (total_len < the_hash_algo->rawsz)
 		return 0; /* say "too short"? */
 
-	the_hash_algo->init_fn(&ctx);
-	the_hash_algo->update_fn(&ctx, data, data_len);
-	the_hash_algo->final_fn(got, &ctx);
+	the_hash_algo->fast_init_fn(&ctx);
+	the_hash_algo->fast_update_fn(&ctx, data, data_len);
+	the_hash_algo->fast_final_fn(got, &ctx);
 
 	return hasheq(got, data + data_len, the_repository->hash_algo);
 }
-- 
2.46.0.425.ge8f5cbd280c

Re: [PATCH 4/4] csum-file.c: use fast SHA-1 implementation when available

[Patrick Steinhardt]

On Sun, Sep 01, 2024 at 12:03:32PM -0400, Taylor Blau wrote:
> Update hashwrite() and friends to use the fast_-variants of hashing
> functions, calling for e.g., "the_hash_algo->fast_update_fn()" instead
> of "the_hash_algo->update_fn()".
> 
> These callers only use the_hash_algo to produce a checksum, which we
> depend on for data integrity, but not for cryptographic purposes, so
> these callers are safe to use the fast (and potentially non-collision
> detecting) SHA-1 implementation.
> 
> To time this, I took a freshly packed copy of linux.git, and ran the
> following with and without the OPENSSL_SHA1_FAST=1 build-knob. Both
> versions were compiled with -O3:
> 
>     $ git for-each-ref --format='%(objectname)' refs/heads refs/tags >in
>     $ valgrind --tool=callgrind ~/src/git/git-pack-objects \
>         --revs --stdout --all-progress --use-bitmap-index <in >/dev/null
> 
> Without OPENSSL_SHA1_FAST=1 (that is, using the collision-detecting
> SHA-1 implementation for both cryptographic and non-cryptographic
> purposes), we spend a significant amount of our instruction count in
> hashwrite():
> 
>     $ callgrind_annotate --inclusive=yes | grep hashwrite | head -n1
>     159,998,868,413 (79.42%)  /home/ttaylorr/src/git/csum-file.c:hashwrite [/home/ttaylorr/src/git/git-pack-objects]
> 
> , and the resulting "clone" takes 19.219 seconds of wall clock time,
> 18.94 seconds of user time and 0.28 seconds of system time.
> 
> Compiling with OPENSSL_SHA1_FAST=1, we spend ~60% fewer instructions in
> hashwrite():
> 
>     $ callgrind_annotate --inclusive=yes | grep hashwrite | head -n1
>      59,164,001,176 (58.79%)  /home/ttaylorr/src/git/csum-file.c:hashwrite [/home/ttaylorr/src/git/git-pack-objects]
> 
> , and generate the resulting "clone" much faster, in only 11.597 seconds
> of wall time, 11.37 seconds of user time, and 0.23 seconds of system
> time, for a ~40% speed-up.

Neat. I knew that SHA1DC was slower, but I certainly didn't expect it to
make such a huge difference.

I of course wish that we just moved on and switched the default to
SHA256, which should provide similar speedups. But that of course
wouldn't help all the projects out there that will use SHA1 for the next
couple decades.

One thing I'm missing is an analysis of users of "csum-file.c" so that
we can assess whether it is safe to switch over this subsystem to use
the fast variant. As far as I can see it's used for packfiles, commit
graphs, the index, packfile reverse indices, MIDXs. None of them strike
me as particularly critical, also because almost all of them would be
created locally by the user anyway.

The only exception are of course packfiles, which get generated by the
remote. Is it possible to generate packfiles with colliding trailer
hashes? And if so, how would the client behave if it was served a
packfile with such a collision?

Patrick

Re: [PATCH 4/4] csum-file.c: use fast SHA-1 implementation when available

[brian m. carlson]

[-- Attachment #1: Type: text/plain, Size: 2978 bytes --]

On 2024-09-02 at 13:41:31, Patrick Steinhardt wrote:
> Neat. I knew that SHA1DC was slower, but I certainly didn't expect it to
> make such a huge difference.
> 
> I of course wish that we just moved on and switched the default to
> SHA256, which should provide similar speedups. But that of course
> wouldn't help all the projects out there that will use SHA1 for the next
> couple decades.

I actually think that not adopting this approach would be a big
incentive to get people to switch, honestly.  We can say, "Need better
performance?  Use SHA-256."  SHA-1 is faster than SHA-256 both when both
are in software or both in hardware (because it does less work per
round, which is part of why it's insecure), and thus this series
actually disincentivizes people from switching because it makes SHA-256
slower by comparison.

We know many people don't care about the security because they're still
using MD5 because it's fast and "it's good enough."  That, by the way,
is absolutely not the opinion of any reputable security or cryptographic
organization.  Similarly, those organizations would say that SHA-1
should also no longer be used, which is a position I would also endorse.

SHA-1 is not going to last another couple decades.  As I mentioned
elsewhere in the thread, someone has already built DES (2^56 work) brute
force hardware for USD 250,000 and is farming it out for as low as USD
20 per crack.  SHA-1 offers 2^61 collision resistance, so a brute force
attack is only 32 times harder (if the operations were completely
equivalent). That means that if someone built a similar machine for
SHA-1 and rented it out, it would probably only cost USD 640 to wreak
havoc on any hosting provider still using SHA-1.[0]  That assumes the
attack doesn't get better, which it probably will.  Note that it took
only 7 years from the first MD5 collision to an attack which runs in
less than a second on a modern computer.

This series also means we have to continue to maintain non-DC versions
of SHA-1, which I had hoped to get rid of.

For those reasons, I'm in general opposed to this series.

> The only exception are of course packfiles, which get generated by the
> remote. Is it possible to generate packfiles with colliding trailer
> hashes? And if so, how would the client behave if it was served a
> packfile with such a collision?

Yes, under the same conditions as colliding any other body text, as I
mentioned elsewhere in the thread.  It would overwrite any existing data
with the same pack hash because we use rename(2).  We would have to use
link(2) and die if the file already existed.

[0] Remember, we die on collisions, so for a push with a colliding
object or pack over HTTP the user would get a 500 error.  Repeat that
push a couple thousand times at 2 a.m. and it'll page the on call
engineer, who I assure you will not be delighted.
-- 
brian m. carlson (they/them or he/him)
Toronto, Ontario, CA

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 262 bytes --]

Re: [PATCH 4/4] csum-file.c: use fast SHA-1 implementation when available

[Taylor Blau]

On Mon, Sep 02, 2024 at 03:41:31PM +0200, Patrick Steinhardt wrote:
> On Sun, Sep 01, 2024 at 12:03:32PM -0400, Taylor Blau wrote:
> > Update hashwrite() and friends to use the fast_-variants of hashing
> > functions, calling for e.g., "the_hash_algo->fast_update_fn()" instead
> > of "the_hash_algo->update_fn()".
> >
> > These callers only use the_hash_algo to produce a checksum, which we
> > depend on for data integrity, but not for cryptographic purposes, so
> > these callers are safe to use the fast (and potentially non-collision
> > detecting) SHA-1 implementation.
> >
> > To time this, I took a freshly packed copy of linux.git, and ran the
> > following with and without the OPENSSL_SHA1_FAST=1 build-knob. Both
> > versions were compiled with -O3:
> >
> >     $ git for-each-ref --format='%(objectname)' refs/heads refs/tags >in
> >     $ valgrind --tool=callgrind ~/src/git/git-pack-objects \
> >         --revs --stdout --all-progress --use-bitmap-index <in >/dev/null
> >
> > Without OPENSSL_SHA1_FAST=1 (that is, using the collision-detecting
> > SHA-1 implementation for both cryptographic and non-cryptographic
> > purposes), we spend a significant amount of our instruction count in
> > hashwrite():
> >
> >     $ callgrind_annotate --inclusive=yes | grep hashwrite | head -n1
> >     159,998,868,413 (79.42%)  /home/ttaylorr/src/git/csum-file.c:hashwrite [/home/ttaylorr/src/git/git-pack-objects]
> >
> > , and the resulting "clone" takes 19.219 seconds of wall clock time,
> > 18.94 seconds of user time and 0.28 seconds of system time.
> >
> > Compiling with OPENSSL_SHA1_FAST=1, we spend ~60% fewer instructions in
> > hashwrite():
> >
> >     $ callgrind_annotate --inclusive=yes | grep hashwrite | head -n1
> >      59,164,001,176 (58.79%)  /home/ttaylorr/src/git/csum-file.c:hashwrite [/home/ttaylorr/src/git/git-pack-objects]
> >
> > , and generate the resulting "clone" much faster, in only 11.597 seconds
> > of wall time, 11.37 seconds of user time, and 0.23 seconds of system
> > time, for a ~40% speed-up.
>
> Neat. I knew that SHA1DC was slower, but I certainly didn't expect it to
> make such a huge difference.

Yeah, I was similarly surprised as you are.

> One thing I'm missing is an analysis of users of "csum-file.c" so that
> we can assess whether it is safe to switch over this subsystem to use
> the fast variant. As far as I can see it's used for packfiles, commit
> graphs, the index, packfile reverse indices, MIDXs. None of them strike
> me as particularly critical, also because almost all of them would be
> created locally by the user anyway.

Right, the only case I believe we care about hash collisions is writing
loose objects, and hashing packed objects from within a packfile. The
loose object write path does not use hashfile(), so it is immune from
these changes.

The path where we read the objects packed in a packfile to determine
their OID is also safe, because that happens in index-pack and does not
use the _fast variants of these functions.

> The only exception are of course packfiles, which get generated by the
> remote. Is it possible to generate packfiles with colliding trailer
> hashes? And if so, how would the client behave if it was served a
> packfile with such a collision?

I think the answer to this is "no", but let's consolidate this
discussion into the sub-thread where brian and I are already chatting
about this to avoid having the discussion in two places.

Thanks,
Taylor

Re: [PATCH 0/4] hash.h: support choosing a separate SHA-1 for non-cryptographic uses

[Junio C Hamano]

Taylor Blau <me@ttaylorr.com> writes:

> After some profiling, I noticed that we spend a significant amount of
> time in hashwrite(), which is not all that surprising. But much of that
> time is wasted in GitHub's infrastructure, since we are using the same
> collision-detecting SHA-1 implementation to produce a trailing checksum
> for the pack which does not need to be cryptographically secure.

Cute.

I wish we can upgrade the file formats so that a writer can choose
the hash algorithm independently from whatever the payload uses.
Most of our use of the tail sum are for files that are consumed
locally in the same repository so nobody shouldn't need to know that
you are using xxHash for the tail sum instead.

Except that the story is not so simple for packfiles, which is named
after the file's tail sum, so you cannot use a hash algorithm of
your choice independently without affecting other folks.  All other
csum-file protected file types are lot smaller than the pack files
to matter, sadly.

Re: [PATCH 0/4] hash.h: support choosing a separate SHA-1 for non-cryptographic uses

[Taylor Blau]

On Sun, Sep 01, 2024 at 08:41:36PM -0700, Junio C Hamano wrote:
> Taylor Blau <me@ttaylorr.com> writes:
>
> > After some profiling, I noticed that we spend a significant amount of
> > time in hashwrite(), which is not all that surprising. But much of that
> > time is wasted in GitHub's infrastructure, since we are using the same
> > collision-detecting SHA-1 implementation to produce a trailing checksum
> > for the pack which does not need to be cryptographically secure.
>
> Cute.
>
> I wish we can upgrade the file formats so that a writer can choose
> the hash algorithm independently from whatever the payload uses.
> Most of our use of the tail sum are for files that are consumed
> locally in the same repository so nobody shouldn't need to know that
> you are using xxHash for the tail sum instead.

Yeah. I would actually like to get here in the long-term, but of course
that change is much larger than this one (the protocol would have to be
adjusted to learn a new "tailsum" capability for callers to negotiate
which checksumming hash function they want to use, etc.).

> Except that the story is not so simple for packfiles, which is named
> after the file's tail sum, so you cannot use a hash algorithm of
> your choice independently without affecting other folks.  All other
> csum-file protected file types are lot smaller than the pack files
> to matter, sadly.

Right.

Thanks,
Taylor

Re: [PATCH 0/4] hash.h: support choosing a separate SHA-1 for non-cryptographic uses

[Junio C Hamano]

Taylor Blau <me@ttaylorr.com> writes:

> Yeah. I would actually like to get here in the long-term, but of course
> that change is much larger than this one (the protocol would have to be
> adjusted to learn a new "tailsum" capability for callers to negotiate
> which checksumming hash function they want to use, etc.).

I dunno.  I was talking about strictly local tail sum, like the ones
in the index file.  You do not have anybody to negotiate about
anything.

We already agreed that the name of a packfile is a different matter.

Re: [PATCH 0/4] hash.h: support choosing a separate SHA-1 for non-cryptographic uses

[brian m. carlson]

[-- Attachment #1: Type: text/plain, Size: 3439 bytes --]

On 2024-09-01 at 16:03:15, Taylor Blau wrote:
> This series adds a build-time knob to allow selecting an alternative
> SHA-1 implementation for non-cryptographic hashing within Git, starting
> with the `hashwrite()` family of functions.
> 
> This series is the result of starting to roll out verbatim multi-pack
> reuse within GitHub's infrastructure. I noticed that on larger
> repositories, it is harder thus far to measure a CPU speed-up on clones
> where multi-pack reuse is enabled.
> 
> After some profiling, I noticed that we spend a significant amount of
> time in hashwrite(), which is not all that surprising. But much of that
> time is wasted in GitHub's infrastructure, since we are using the same
> collision-detecting SHA-1 implementation to produce a trailing checksum
> for the pack which does not need to be cryptographically secure.

Hmm, I'm not sure this is the case.  Let's consider the case where SHA-1
becomes as easy to collide as MD4, which requires less than 2 hash
operations for a collision, in which case we can assume that it's
trivial, because eventually we expect that will happen with advances in
technology.

So in that case, we believe that an attacker who knows what's in a pack
file and can collide one or more of the objects can create another
packfile with a different, colliding object and cause the pack contents
to be the same.  Because we use the pack file hash as the name of the
pack and we use rename(2), which ignores whether the destination exists,
that means we have to assume that eventually an attacker will be able to
overwrite one pack file with another with different contents without
being detected simply by pushing a new pack into the repository.

Even if we assume that SHA-1 attacks only become as easy as MD5 attacks,
the RapidSSL exploits[0][1] demonstrate that an attacker can create
collisions based on predictable outputs even with imprecise feedback. We
know SHA-1 is quite weak, so it could actually be quite soon that
someone finds an improvement in attacking the algorithm.  Note that
computing 2^56 DES keys by brute force costs about $20 from cloud
providers[2], and SHA-1 provides only 2^61 collision security, so a
small improvement would probably make this pretty viable to attack on
major providers with dedicated hardware.

This is actually worse on some providers where the operations tend to be
single threaded.  In those situations, there is no nondeterminism from
threads to make packs slightly different, and thus it would be extremely
easy to create such collisions predictably based on what an appropriate
upstream version of Git does.

So I don't think we can accurately say that cryptographic security isn't
needed here.  If we need a unique name that an attacker cannot control
and there are negative consequences (such as data loss) from the
attacker being able to control the name, then we need cryptographic
security here, which would imply a collision-detecting SHA-1 algorithm.

We could avoid this problem if we used link(2) and unlink(2) to avoid
renaming over existing pack files, though, similar to loose objects.
We'd need to not use the new fast algorithm unless core.createObject is
set to "link", though.

[0] https://en.wikipedia.org/wiki/MD5
[1] https://www.win.tue.nl/hashclash/rogue-ca/
[2] https://crack.sh/
-- 
brian m. carlson (they/them or he/him)
Toronto, Ontario, CA

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 262 bytes --]

Re: [PATCH 0/4] hash.h: support choosing a separate SHA-1 for non-cryptographic uses

[Taylor Blau]

On Mon, Sep 02, 2024 at 02:08:25PM +0000, brian m. carlson wrote:
> On 2024-09-01 at 16:03:15, Taylor Blau wrote:
> > This series adds a build-time knob to allow selecting an alternative
> > SHA-1 implementation for non-cryptographic hashing within Git, starting
> > with the `hashwrite()` family of functions.
> >
> > This series is the result of starting to roll out verbatim multi-pack
> > reuse within GitHub's infrastructure. I noticed that on larger
> > repositories, it is harder thus far to measure a CPU speed-up on clones
> > where multi-pack reuse is enabled.
> >
> > After some profiling, I noticed that we spend a significant amount of
> > time in hashwrite(), which is not all that surprising. But much of that
> > time is wasted in GitHub's infrastructure, since we are using the same
> > collision-detecting SHA-1 implementation to produce a trailing checksum
> > for the pack which does not need to be cryptographically secure.
>
> Hmm, I'm not sure this is the case.  Let's consider the case where SHA-1
> becomes as easy to collide as MD4, which requires less than 2 hash
> operations for a collision, in which case we can assume that it's
> trivial, because eventually we expect that will happen with advances in
> technology.

I'm not sure this attack is possible as you described.

We still run any packs through index-pack before landing them in
$GIT_DIR/objects/pack, and index-pack still uses the collision-detecting
SHA-1 implementation (if the repository uses SHA-1 and Git was compiled
with it).

So if I were a malicious attacker trying to compromise data on a forge,
I would have to first (a) know the name of some pack that I was trying
to collide, then (b) create a pack which collides with that one before
actually pushing it. (b) seems difficult to impossible to execute
(certainly today, maybe ever) because the attacker only controls the
object contents within the pack, but can't adjust the pack header,
object headers, compression, etc.

But even if the attacker could do all of that, the remote still needs to
index that pack, and while checksumming the pack, it would notice the
collision (or SHA-1 mismatch) and reject the pack by die()-ing either
way. (AFAICT, this all happens in
builtin/index-pack.c::parse_pack_objects()).

> So in that case, we believe that an attacker who knows what's in a pack
> file and can collide one or more of the objects can create another
> packfile with a different, colliding object and cause the pack contents
> to be the same.  Because we use the pack file hash as the name of the
> pack and we use rename(2), which ignores whether the destination exists,
> that means we have to assume that eventually an attacker will be able to
> overwrite one pack file with another with different contents without
> being detected simply by pushing a new pack into the repository.

Right... but I think we would die() before we attempt to rename() the
pack into place as above.

Thanks,
Taylor

Re: [PATCH 0/4] hash.h: support choosing a separate SHA-1 for non-cryptographic uses

[Junio C Hamano]

Taylor Blau <me@ttaylorr.com> writes:

> But even if the attacker could do all of that, the remote still needs to
> index that pack, and while checksumming the pack, it would notice the
> collision (or SHA-1 mismatch) and reject the pack by die()-ing either
> way. (AFAICT, this all happens in
> builtin/index-pack.c::parse_pack_objects()).

The hosting side writes a packfile and computes the tail sum once.
You force the clients that clone or fetch validate the tail sum.
Usually clients outnumber the hoster by large orders of magnitude.

That sounds like you are optimizing for a wrong side, but it does
point at another aspect of this problem.

Even without limiting ourselves to the tail sum, our uses of the
hash function fall into two categories, ones that do not have to be
overly cautious (i.e., when we are generating data and computing the
hash over that data), and the others that we do want to be paranoid
(i.e., when we receive check-summed data from outside world and
suspect that the data was generated by an adversary).

Re: [PATCH 0/4] hash.h: support choosing a separate SHA-1 for non-cryptographic uses

[brian m. carlson]

[-- Attachment #1: Type: text/plain, Size: 1877 bytes --]

On 2024-09-03 at 19:47:39, Taylor Blau wrote:
> We still run any packs through index-pack before landing them in
> $GIT_DIR/objects/pack, and index-pack still uses the collision-detecting
> SHA-1 implementation (if the repository uses SHA-1 and Git was compiled
> with it).
> 
> So if I were a malicious attacker trying to compromise data on a forge,
> I would have to first (a) know the name of some pack that I was trying
> to collide, then (b) create a pack which collides with that one before
> actually pushing it. (b) seems difficult to impossible to execute
> (certainly today, maybe ever) because the attacker only controls the
> object contents within the pack, but can't adjust the pack header,
> object headers, compression, etc.

Packing single-threaded is deterministic in my tests, so it would seem
that this is possible, even if inconvenient or difficult to execute.
It's not very hard to get access to the configuration a forge is using
either because it's open source or open core, or just from getting the
on-premises version's configuration, so we have to assume that the
attacker knows the configuration, and we also can determine what packs
are on the server side if we've pushed all of the objects ourselves.

> But even if the attacker could do all of that, the remote still needs to
> index that pack, and while checksumming the pack, it would notice the
> collision (or SHA-1 mismatch) and reject the pack by die()-ing either
> way. (AFAICT, this all happens in
> builtin/index-pack.c::parse_pack_objects()).

If we're certain that we'll always index the pack, then I agree we would
detect this at that point, and so it would probably be safe.  As you and
I discussed elsewhere, I'm not the expert on the pack code, so I'll
defer to your analysis here.
-- 
brian m. carlson (they/them or he/him)
Toronto, Ontario, CA

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 262 bytes --]

Re: [PATCH 0/4] hash.h: support choosing a separate SHA-1 for non-cryptographic uses

[Jeff King]

On Tue, Sep 03, 2024 at 03:47:39PM -0400, Taylor Blau wrote:

> > Hmm, I'm not sure this is the case.  Let's consider the case where SHA-1
> > becomes as easy to collide as MD4, which requires less than 2 hash
> > operations for a collision, in which case we can assume that it's
> > trivial, because eventually we expect that will happen with advances in
> > technology.
> 
> I'm not sure this attack is possible as you described.
> 
> We still run any packs through index-pack before landing them in
> $GIT_DIR/objects/pack, and index-pack still uses the collision-detecting
> SHA-1 implementation (if the repository uses SHA-1 and Git was compiled
> with it).

I agree this is not a problem with your series as it is, but I think in
the long run we'd want to switch over index-pack, too, for two reasons:

  1. It could benefit from the same speedups on the receiving side that
     your patches give on the sending side (though the difference is
     less noticeable there, because we're also hashing the expanded
     contents).

  2. We'll have to do so if switch to a non-cryptographic hash (as is
     discussed elsewhere in this thread), since the two obviously have
     to match.

So let's imagine for a moment that we make that change.

I don't think you can smuggle any duplicate objects this way. We'll
still index each individual object using sha1dc, so any attempts to
collide there will be caught. You'd need totally different objects that
are in a packfile that happens to hash to the same checksum. And then
since the receiver is the one computing the object id of those objects,
they won't match (modulo some racing, which I'll get to).

But I do think you could do a denial-of-service attack by corrupting the
receiving repo. Imagine that:

  1. I (somehow) know you have a pack with hash XYZ, and thus that
     you'll be storing objects/pack/pack-XYZ.pack.

  2. I generate a new, valid pack that contains 100 random objects
     (or enough to defeat transfer.unpackLimit). I mutate that objects'
     contents until my new pack also has hash XYZ.

  3. I send you that pack (e.g., via push if you're a server, or by
     manipulating you into fetch if you're a client). You index the
     pack, see that it should be called pack-XYZ.pack, and then rename()
     it on top of your existing file.

Now you've just lost access to all of those objects, and your repo is
broken.

We should be able to simulate this in practice. First, let's weaken the
"fast" sha1 such that it retains only the final two bits:

diff --git a/sha1/openssl.h b/sha1/openssl.h
index 1038af47da..f0d5c59c43 100644
--- a/sha1/openssl.h
+++ b/sha1/openssl.h
@@ -32,6 +32,8 @@ static inline void openssl_SHA1_Final(unsigned char *digest,
 {
 	EVP_DigestFinal_ex(ctx->ectx, digest, NULL);
 	EVP_MD_CTX_free(ctx->ectx);
+	memset(digest, 0, 19);
+	digest[19] &= 0x3;
 }
 
 static inline void openssl_SHA1_Clone(struct openssl_SHA1_CTX *dst,

Now we can build with OPENSSL_SHA1_FAST=1, and use the result (which I
call git.compile below) to repack some victim repo:

  # use regular Git to clone first
  git clone --no-local --bare /some/repo victim.git

  # and now use our weak hash to repack; you should end up with
  # pack-000000...003.pack or similar
  git.compile -C victim.git repack -ad

We can even fsck it, if we teach the fsck code to use our weak hash (as
an aside, I think it is weird that fsck has its own hash verification;
it should probably be relying on verify-pack for this, but I haven't dug
into this in a while and IIRC there are some complications):

diff --git a/pack-check.c b/pack-check.c
index e883dae3f2..1b80616c70 100644
--- a/pack-check.c
+++ b/pack-check.c
@@ -67,7 +67,7 @@ static int verify_packfile(struct repository *r,
 	if (!is_pack_valid(p))
 		return error("packfile %s cannot be accessed", p->pack_name);
 
-	r->hash_algo->init_fn(&ctx);
+	r->hash_algo->fast_init_fn(&ctx);
 	do {
 		unsigned long remaining;
 		unsigned char *in = use_pack(p, w_curs, offset, &remaining);
@@ -76,9 +76,9 @@ static int verify_packfile(struct repository *r,
 			pack_sig_ofs = p->pack_size - r->hash_algo->rawsz;
 		if (offset > pack_sig_ofs)
 			remaining -= (unsigned int)(offset - pack_sig_ofs);
-		r->hash_algo->update_fn(&ctx, in, remaining);
+		r->hash_algo->fast_update_fn(&ctx, in, remaining);
 	} while (offset < pack_sig_ofs);
-	r->hash_algo->final_fn(hash, &ctx);
+	r->hash_algo->fast_final_fn(hash, &ctx);
 	pack_sig = use_pack(p, w_curs, pack_sig_ofs, NULL);
 	if (!hasheq(hash, pack_sig, the_repository->hash_algo))
 		err = error("%s pack checksum mismatch",

And now let's teach index-pack to use the weak hash, too:

diff --git a/builtin/index-pack.c b/builtin/index-pack.c
index fd968d673d..de99405880 100644
--- a/builtin/index-pack.c
+++ b/builtin/index-pack.c
@@ -284,7 +284,7 @@ static void flush(void)
 	if (input_offset) {
 		if (output_fd >= 0)
 			write_or_die(output_fd, input_buffer, input_offset);
-		the_hash_algo->update_fn(&input_ctx, input_buffer, input_offset);
+		the_hash_algo->fast_update_fn(&input_ctx, input_buffer, input_offset);
 		memmove(input_buffer, input_buffer + input_offset, input_len);
 		input_offset = 0;
 	}
@@ -357,7 +357,7 @@ static const char *open_pack_file(const char *pack_name)
 		output_fd = -1;
 		nothread_data.pack_fd = input_fd;
 	}
-	the_hash_algo->init_fn(&input_ctx);
+	the_hash_algo->fast_init_fn(&input_ctx);
 	return pack_name;
 }
 
@@ -1202,9 +1202,9 @@ static void parse_pack_objects(unsigned char *hash)
 
 	/* Check pack integrity */
 	flush();
-	the_hash_algo->init_fn(&tmp_ctx);
-	the_hash_algo->clone_fn(&tmp_ctx, &input_ctx);
-	the_hash_algo->final_fn(hash, &tmp_ctx);
+	the_hash_algo->fast_init_fn(&tmp_ctx);
+	the_hash_algo->fast_clone_fn(&tmp_ctx, &input_ctx);
+	the_hash_algo->fast_final_fn(hash, &tmp_ctx);
 	if (!hasheq(fill(the_hash_algo->rawsz), hash, the_repository->hash_algo))
 		die(_("pack is corrupted (SHA1 mismatch)"));
 	use(the_hash_algo->rawsz);

Now the code is all set. We need to adjust two things in the victim
repo:

  # we could just send 100+ fake objects from the client, but setting this to
  # "1" makes our attack loop simpler
  git -C victim.git config transfer.unpackLimit 1

  # Pushing happens into a quarantine area. And then when we move the
  # files out of quarantine, we do it with finalize_object_file(), which
  # does the usual link collision detection. But:
  #
  #   1. That doesn't happen for fetches, which are done directly in
  #      the pack directory, so the rename() done by index-pack applies
  #      there.
  #
  #   2. If the link() doesn't work for any reason other than EEXIST,
  #      we'll retry it as a rename(). So depending on your filesystem,
  #      this might be triggerable even for push.
  #
  # To simulate the worst case, we'll manually force the push into
  # rename mode.
  git -C victim.git config core.createObject rename

And now we can attack, like so:

  git init evil
  cd evil
  while git.compile -C ../victim.git fsck; do
          ls -l ../victim.git/objects/pack/
          git.compile commit --allow-empty -m foo
          git.compile push ../victim.git HEAD:foo
  done
  ls -l ../victim.git/objects/pack/

It's random whether we'll collide, but since there are only 4
possibilities (from 2 bits), it happens within a couple attempts.
Obviously sha1 is stronger than that, but I think the point is to
prepare for a world where collisions are cheap and easy to produce
(whether because sha1 gets more broken, or because we start using a
non-cryptographic hash).

So I do think there are problems in this general path, even though your
patch is not (yet) exposing them (although it would be easy to do so
accidentally; I was actually moderately surprised that index-pack is not
relying on the hashfile API already).

Probably the solution is:

  - renaming packfiles into place should use finalize_object_file() to
    avoid collisions.  That happens for tmp-objdir migration already,
    but we should do it more directly in index-pack (and maybe even
    pack-objects). And possibly we should implement an actual
    byte-for-byte comparison if we think we saw a collision, rather than
    just assuming that the write was effectively a noopi (see the FIXME
    in that function). That becomes more important if the checksum gets
    more likely to collide accidentally (we essentially ignore the
    possibility that sha1 would ever do so).

  - possibly object_creation_mode should have a more strict setting that
    refuses to fall back to renames. Or alternatively, we should do our
    own check for existence when falling back to a rename() in
    finalize_object_file().


And finally, I mentioned races earlier. I didn't try to reproduce it,
but I suspect there could also be a race like:

  - you have pack-XYZ.pack and pack-XYZ.idx

  - attacker generates evil pack-XYZ.pack as above (and assuming we have
    no fixed anything as I suggested above, and they really can replace
    the receiver's copy).

  - at some moment we will have moved pack-XYZ.pack into place, but not
    yet the matching idx. So we'll have the old idx and the new pack. An
    object lookup at that moment could cause us to find the object using
    the old idx, but then get the data out of the new pack file,
    replacing real data with the attacker's data. It's a pretty small
    window, but probably possible with enough simultaneous reading
    processes. Not something you'd probably want to spend $40k
    generating a collision for, but if we used a weak enough checksum,
    then attempts become cheap.

So I think we really do need to address this to prefer local data. At
which point it should be safe to use a much weaker checksum. But IMHO it
is still worth having a "fast" SHA1. Even if the future is SHA256 repos
or xxHash pack trailers, older clients will still use SHA1.

-Peff

Re: [PATCH 0/4] hash.h: support choosing a separate SHA-1 for non-cryptographic uses

[Junio C Hamano]

Jeff King <peff@peff.net> writes:

> Probably the solution is:
>
>   - renaming packfiles into place should use finalize_object_file() to
>     avoid collisions.  That happens for tmp-objdir migration already,
>     but we should do it more directly in index-pack (and maybe even
>     pack-objects). And possibly we should implement an actual
>     byte-for-byte comparison if we think we saw a collision, rather than
>     just assuming that the write was effectively a noopi (see the FIXME
>     in that function). That becomes more important if the checksum gets
>     more likely to collide accidentally (we essentially ignore the
>     possibility that sha1 would ever do so).

Yes.  I somehow mistakenly thought that Taylor analized the code
path when brian raised the "we rename over, overwriting existing
files" and I included fixing it as one of the steps necessary to
safely switch the tail sum to weaker and faster hash, but after
reading the thread again, it seems that I was hallucinating X-<.
This needs to be corrected.

>   - possibly object_creation_mode should have a more strict setting that
>     refuses to fall back to renames. Or alternatively, we should do our
>     own check for existence when falling back to a rename() in
>     finalize_object_file().

True, too.

>   - at some moment we will have moved pack-XYZ.pack into place, but not
>     yet the matching idx. So we'll have the old idx and the new pack. An
>     object lookup at that moment could cause us to find the object using
>     the old idx, but then get the data out of the new pack file,
>     replacing real data with the attacker's data. It's a pretty small
>     window, but probably possible with enough simultaneous reading
>     processes. Not something you'd probably want to spend $40k
>     generating a collision for, but if we used a weak enough checksum,
>     then attempts become cheap.

This reminds me why we changed the hash we use to name packfiles; we
used to use "hash of sorted object names contained in the pack", but
that would mean a (forced) repack of a sole pack of a fully packed
repository can create a packfile with contents and object layout
different from the original but with the same name, creating this
exact race to yourself without involving any evil attacker.  We of
course use the hash of the actual pack data stream these days, and
that would avoid this problem. 

It is funny to compare this with the reason why we switched how we
name individual objects in a very early part in the history.  We
used to name an object after the hash value of _compressed_ object
header plus payload, but that obviously means different compression
level (or improvement of the compressor implementation) would give
different names to the same contents, and that is why we swapped the
order to use the hash value of the object header plus payload before
compression.  Of course, that _requires_ us to avoid overwriting an
existing file to foil collision attack.  That brings us back to the
topic here ;-).

> So I think we really do need to address this to prefer local data. At
> which point it should be safe to use a much weaker checksum. But IMHO it
> is still worth having a "fast" SHA1. Even if the future is SHA256 repos
> or xxHash pack trailers, older clients will still use SHA1.

Yup. 100% agreed.

Thanks.

Re: [PATCH 0/4] hash.h: support choosing a separate SHA-1 for non-cryptographic uses

[Taylor Blau]

On Thu, Sep 05, 2024 at 08:41:16AM -0700, Junio C Hamano wrote:
> Jeff King <peff@peff.net> writes:
>
> > Probably the solution is:
> >
> >   - renaming packfiles into place should use finalize_object_file() to
> >     avoid collisions.  That happens for tmp-objdir migration already,
> >     but we should do it more directly in index-pack (and maybe even
> >     pack-objects). And possibly we should implement an actual
> >     byte-for-byte comparison if we think we saw a collision, rather than
> >     just assuming that the write was effectively a noopi (see the FIXME
> >     in that function). That becomes more important if the checksum gets
> >     more likely to collide accidentally (we essentially ignore the
> >     possibility that sha1 would ever do so).
>
> Yes.  I somehow mistakenly thought that Taylor analized the code
> path when brian raised the "we rename over, overwriting existing
> files" and I included fixing it as one of the steps necessary to
> safely switch the tail sum to weaker and faster hash, but after
> reading the thread again, it seems that I was hallucinating X-<.
> This needs to be corrected.

Just to make sure I understand you here... this is talking about a
prerequisite step for switching index-pack to use a faster hash, right?

If so, I agree, but would note that this series does not yet switch
index-pack to use the non-collision detecting SHA-1 implementation when
available, so that would not be a prerequisite for merging this series.

Thanks,
Taylor

Re: [PATCH 0/4] hash.h: support choosing a separate SHA-1 for non-cryptographic uses

[Junio C Hamano]

Taylor Blau <me@ttaylorr.com> writes:

> If so, I agree, but would note that this series does not yet switch
> index-pack to use the non-collision detecting SHA-1 implementation when
> available, so that would not be a prerequisite for merging this series.

Hmph, I am confused.  It needs to be corrected in order to address
collisions of the tail sum Peff raised, as no longer checked the
tail sum with SHA1DC but with "fast" SHA-1.

Doesn't it make it a prerequisite?

Re: [PATCH 0/4] hash.h: support choosing a separate SHA-1 for non-cryptographic uses

[Taylor Blau]

On Thu, Sep 05, 2024 at 09:51:00AM -0700, Junio C Hamano wrote:
> Taylor Blau <me@ttaylorr.com> writes:
>
> > If so, I agree, but would note that this series does not yet switch
> > index-pack to use the non-collision detecting SHA-1 implementation when
> > available, so that would not be a prerequisite for merging this series.
>
> Hmph, I am confused.  It needs to be corrected in order to address
> collisions of the tail sum Peff raised, as no longer checked the
> tail sum with SHA1DC but with "fast" SHA-1.

Peff's mail supposes that we have already modified index-pack to use the
non-collision detecting SHA-1 implementation. But this series does not
do that, so I don't think we have an issue to address here.

In a hypothetical future series where we do modify index-pack to use the
_FAST SHA-1 implementation, then we would need to address the issue that
Peff raised first as a prerequisite.

Thanks,
Taylor

Re: [PATCH 0/4] hash.h: support choosing a separate SHA-1 for non-cryptographic uses

[Taylor Blau]

On Thu, Sep 05, 2024 at 01:04:34PM -0400, Taylor Blau wrote:
> On Thu, Sep 05, 2024 at 09:51:00AM -0700, Junio C Hamano wrote:
> > Taylor Blau <me@ttaylorr.com> writes:
> >
> > > If so, I agree, but would note that this series does not yet switch
> > > index-pack to use the non-collision detecting SHA-1 implementation when
> > > available, so that would not be a prerequisite for merging this series.
> >
> > Hmph, I am confused.  It needs to be corrected in order to address
> > collisions of the tail sum Peff raised, as no longer checked the
> > tail sum with SHA1DC but with "fast" SHA-1.
>
> Peff's mail supposes that we have already modified index-pack to use the
> non-collision detecting SHA-1 implementation. But this series does not
> do that, so I don't think we have an issue to address here.
>
> In a hypothetical future series where we do modify index-pack to use the
> _FAST SHA-1 implementation, then we would need to address the issue that
> Peff raised first as a prerequisite.

I verified that this was the case by applying only the following to my
series:

--- 8< ---
diff --git a/sha1/openssl.h b/sha1/openssl.h
index 1038af47da..f0d5c59c43 100644
--- a/sha1/openssl.h
+++ b/sha1/openssl.h
@@ -32,6 +32,8 @@ static inline void openssl_SHA1_Final(unsigned char *digest,
 {
 	EVP_DigestFinal_ex(ctx->ectx, digest, NULL);
 	EVP_MD_CTX_free(ctx->ectx);
+	memset(digest, 0, 19);
+	digest[19] &= 0x3;
 }

 static inline void openssl_SHA1_Clone(struct openssl_SHA1_CTX *dst,
--- >8 ---

and then creating a victim.git repository (which in my case was born
from git.git) and then repacking to produce the following state:

    $ ls -la victim.git/objects/pack
    total 262704
    drwxr-xr-x 2 ttaylorr ttaylorr      4096 Sep  5 13:45 .
    drwxr-xr-x 4 ttaylorr ttaylorr      4096 Sep  5 13:46 ..
    -r--r--r-- 1 ttaylorr ttaylorr   3306804 Sep  5 13:45 pack-0000000000000000000000000000000000000003.bitmap
    -r--r--r-- 1 ttaylorr ttaylorr  15588224 Sep  5 13:44 pack-0000000000000000000000000000000000000003.idx
    -r--r--r-- 1 ttaylorr ttaylorr 247865480 Sep  5 13:44 pack-0000000000000000000000000000000000000003.pack
    -r--r--r-- 1 ttaylorr ttaylorr   2226788 Sep  5 13:44 pack-0000000000000000000000000000000000000003.rev

Then I set up an "evil" repository like in Peff's recipe and started
repeatedly pushing. fsck is slow here, so the loop is just "while true",
but it doesn't matter that we're not fscking the victim repository since
I'll show in a second that it's not corrupted to begin with.

Running this loop:

  $ while true
    do
      ls -l ../victim.git/objects/pack/
      git.compile commit --allow-empty -m foo
      git.compile push ../victim.git HEAD:foo
    done
  $ ls -l ../victim.git/objects/pack/

, fails very quickly and produces the following:

    [main 727346d] foo
    Enumerating objects: 12, done.
    Counting objects: 100% (12/12), done.
    Delta compression using up to 20 threads
    Compressing objects: 100% (11/11), done.
    Writing objects: 100% (12/12), 779 bytes | 779.00 KiB/s, done.
    Total 12 (delta 10), reused 0 (delta 0), pack-reused 0 (from 0)
    remote: fatal: final sha1 did not match
    error: remote unpack failed: unpack-objects abnormal exit
    To ../victim.git
     ! [remote rejected] HEAD -> foo (unpacker error)
    error: failed to push some refs to '../victim.git'

The victim repository rightly rejects the push, since even though the
evil repository generated a pack with a colliding checksum value, the
victim repository validated it using the collision-detecting /
non-broken SHA-1 implementation and rejected the pack appropriately.

Of course, if index-pack were updated to use the non-collision detecting
implementation of SHA-1 when compiled using one of the _FAST knobs,
*and* we did blindly updated index-pack to use the _fast variants
without doing anything else in Peff's mail, then we would have
corruption.

But I think the point of Peff's mail is to illustrate that this is only
a problem in a world where index-pack uses the _fast SHA-1
implementation, but does not have any additional protections in place.

Thanks,
Taylor

Re: [PATCH 0/4] hash.h: support choosing a separate SHA-1 for non-cryptographic uses

[Taylor Blau]

On Thu, Sep 05, 2024 at 01:51:10PM -0400, Taylor Blau wrote:
> , fails very quickly and produces the following:
>
>     [main 727346d] foo
>     Enumerating objects: 12, done.
>     Counting objects: 100% (12/12), done.
>     Delta compression using up to 20 threads
>     Compressing objects: 100% (11/11), done.
>     Writing objects: 100% (12/12), 779 bytes | 779.00 KiB/s, done.
>     Total 12 (delta 10), reused 0 (delta 0), pack-reused 0 (from 0)
>     remote: fatal: final sha1 did not match
>     error: remote unpack failed: unpack-objects abnormal exit
>     To ../victim.git
>      ! [remote rejected] HEAD -> foo (unpacker error)
>     error: failed to push some refs to '../victim.git'

I didn't set transfer.unpackLimit to zero here, which is why this says
"(unpacker error)". But if I did remember, it would instead say:

    remote: fatal: pack is corrupted (SHA1 mismatch)
    error: remote unpack failed: index-pack abnormal exit
    To ../victim.git
     ! [remote rejected] HEAD -> foo2 (unpacker error)
    error: failed to push some refs to '../victim.git'

So we're safe here whether or not you use the unpackLimit setting.

Thanks,
Taylor

Re: [PATCH 0/4] hash.h: support choosing a separate SHA-1 for non-cryptographic uses

[Jeff King]

On Thu, Sep 05, 2024 at 01:04:34PM -0400, Taylor Blau wrote:

> On Thu, Sep 05, 2024 at 09:51:00AM -0700, Junio C Hamano wrote:
> > Taylor Blau <me@ttaylorr.com> writes:
> >
> > > If so, I agree, but would note that this series does not yet switch
> > > index-pack to use the non-collision detecting SHA-1 implementation when
> > > available, so that would not be a prerequisite for merging this series.
> >
> > Hmph, I am confused.  It needs to be corrected in order to address
> > collisions of the tail sum Peff raised, as no longer checked the
> > tail sum with SHA1DC but with "fast" SHA-1.
> 
> Peff's mail supposes that we have already modified index-pack to use the
> non-collision detecting SHA-1 implementation. But this series does not
> do that, so I don't think we have an issue to address here.

Yes, this is correct. You are modifying only the writing side (which of
course would be under attacker control in this scenario anyway). So we
are only getting the benefit there, but without any additional threat on
the reading side.

But I'm not sure how comfortable I am leaving us in that state, even if
it is by itself OK. It feels fragile, and we're a small step away from
somebody accidentally using the "fast" variant to do reading.

If it's possible to fix the overwrite issue without too much code (and I
think it probably is), then that leaves us in a much safer spot, even
with what's in your series. And the fact that it lets us speed up the
reading side _and_ opens the door to possible alternate-hash
improvements is the cherry on top.

  Side note: I do really like the xxHash direction in general, but I
  think we may be underestimating the difficulty. Obviously it needs a
  protocol extension for sending packfiles, but what about other
  cross-repo access? E.g., dumb http blindly downloads the packfiles.
  How does it learn which hash is in use?

-Peff

Re: [PATCH 0/4] hash.h: support choosing a separate SHA-1 for non-cryptographic uses

[Junio C Hamano]

Jeff King <peff@peff.net> writes:

> But I'm not sure how comfortable I am leaving us in that state, even if
> it is by itself OK. It feels fragile, and we're a small step away from
> somebody accidentally using the "fast" variant to do reading.

Yup, avoiding to rename over an existing files makes the gives us a
lot more defensive posture.

>   Side note: I do really like the xxHash direction in general, but I
>   think we may be underestimating the difficulty. Obviously it needs a
>   protocol extension for sending packfiles, but what about other
>   cross-repo access? E.g., dumb http blindly downloads the packfiles.
>   How does it learn which hash is in use?

We need to make the file contents a bit more self describing
regardless.  A dumb http clone should look more or less like

 - download "packfiles"
 - download "info/refs"

 - run verify-pack on the packfiles
 - create an empty repository
 - move the pack we downloaded to .git/objects/pack/
 - populate refs from downloaded info/refs

 - make sure the tips of refs are all connected.

but as we recently discussed how verify-pack and index-pack assumes
the hash algorithm in the receiving repository, we need to give a
bit more clue in packfiles themselves what hash algorithms they use,
etc.

[PATCH v2 0/4] hash.h: support choosing a separate SHA-1 for non-cryptographic uses

[Taylor Blau]

This series adds a build-time knob to allow selecting an alternative
SHA-1 implementation for non-cryptographic hashing within Git, starting
with the `hashwrite()` family of functions.

This version is a small reroll from the original round which addresses a
handful of suggestions made during review, and also fixes compiling with
OPENSSL_SHA1_FAST with older versions of OpenSSL (having
OPENSSL_API_LEVEL < 3).

Otherwise, the series is unchanged from the first round. But as always,
a range-diff is included below for convenience.

Thanks in advance for your review!

Taylor Blau (4):
  sha1: do not redefine `platform_SHA_CTX` and friends
  hash.h: scaffolding for _fast hashing variants
  Makefile: allow specifying a SHA-1 for non-cryptographic uses
  csum-file.c: use fast SHA-1 implementation when available

 Makefile          | 25 ++++++++++++++++
 block-sha1/sha1.h |  2 ++
 csum-file.c       | 18 ++++++------
 hash.h            | 72 +++++++++++++++++++++++++++++++++++++++++++++++
 object-file.c     | 42 +++++++++++++++++++++++++++
 sha1/openssl.h    |  2 ++
 sha1dc_git.h      |  3 ++
 7 files changed, 155 insertions(+), 9 deletions(-)

Range-diff against v1:
1:  e7cd23bf4c = 1:  e7cd23bf4c sha1: do not redefine `platform_SHA_CTX` and friends
2:  6ac6f934c3 ! 2:  3b5f21e4a6 hash.h: scaffolding for _fast hashing variants
    @@ hash.h
      #endif

     +#ifndef platform_SHA_CTX_fast
    -+#define platform_SHA_CTX_fast	  platform_SHA_CTX
    -+#define platform_SHA1_Init_fast	  platform_SHA1_Init
    -+#define platform_SHA1_Update_fast platform_SHA1_Update
    -+#define platform_SHA1_Final_fast  platform_SHA1_Final
    -+#ifdef platform_SHA1_Clone
    -+#define platform_SHA1_Clone_fast  platform_SHA1_Clone
    -+#endif
    ++#  define platform_SHA_CTX_fast     platform_SHA_CTX
    ++#  define platform_SHA1_Init_fast   platform_SHA1_Init
    ++#  define platform_SHA1_Update_fast platform_SHA1_Update
    ++#  define platform_SHA1_Final_fast  platform_SHA1_Final
    ++#  ifdef platform_SHA1_Clone
    ++#    define platform_SHA1_Clone_fast platform_SHA1_Clone
    ++#  endif
     +#endif
     +
      #define git_SHA_CTX		platform_SHA_CTX
    @@ hash.h
      #define git_SHA1_Clone	platform_SHA1_Clone
      #endif
     +#ifdef platform_SHA1_Clone_fast
    -+#define git_SHA1_Clone_fast	platform_SHA1_Clone_fast
    ++#  define git_SHA1_Clone_fast platform_SHA1_Clone_fast
     +#endif

      #ifndef platform_SHA256_CTX
    @@ hash.h: struct git_hash_algo {
      	/* The hash finalization function for object IDs. */
      	git_hash_final_oid_fn final_oid_fn;

    -+	/* The fast hash initialization function. */
    ++	/* The fast / non-cryptographic hash initialization function. */
     +	git_hash_init_fn fast_init_fn;
     +
    -+	/* The fast hash context cloning function. */
    ++	/* The fast / non-cryptographic hash context cloning function. */
     +	git_hash_clone_fn fast_clone_fn;
     +
    -+	/* The fast hash update function. */
    ++	/* The fast / non-cryptographic hash update function. */
     +	git_hash_update_fn fast_update_fn;
     +
    -+	/* The fast hash finalization function. */
    ++	/* The fast / non-cryptographic hash finalization function. */
     +	git_hash_final_fn fast_final_fn;
     +
    -+	/* The fast hash finalization function for object IDs. */
    ++	/* The fast / non-cryptographic hash finalization function. */
     +	git_hash_final_oid_fn fast_final_oid_fn;
     +
      	/* The OID of the empty tree. */
3:  682e4c2cc3 ! 3:  02764de139 Makefile: allow specifying a SHA-1 for non-cryptographic uses
    @@ hash.h
      #endif

     +#if defined(SHA1_APPLE_FAST)
    -+#include <CommonCrypto/CommonDigest.h>
    -+#define platform_SHA_CTX_fast CC_SHA1_CTX
    -+#define platform_SHA1_Init_fast CC_SHA1_Init
    -+#define platform_SHA1_Update_fast CC_SHA1_Update
    -+#define platform_SHA1_Final_fast CC_SHA1_Final
    ++#  include <CommonCrypto/CommonDigest.h>
    ++#  define platform_SHA_CTX_fast CC_SHA1_CTX
    ++#  define platform_SHA1_Init_fast CC_SHA1_Init
    ++#  define platform_SHA1_Update_fast CC_SHA1_Update
    ++#  define platform_SHA1_Final_fast CC_SHA1_Final
     +#elif defined(SHA1_OPENSSL_FAST)
     +#  include <openssl/sha.h>
     +#  if defined(OPENSSL_API_LEVEL) && OPENSSL_API_LEVEL >= 3
     +#    define SHA1_NEEDS_CLONE_HELPER_FAST
     +#    include "sha1/openssl.h"
    ++#    define platform_SHA_CTX_fast openssl_SHA1_CTX
    ++#    define platform_SHA1_Init_fast openssl_SHA1_Init
    ++#    define platform_SHA1_Clone_fast openssl_SHA1_Clone
    ++#    define platform_SHA1_Update_fast openssl_SHA1_Update
    ++#    define platform_SHA1_Final_fast openssl_SHA1_Final
    ++#  else
    ++#    define platform_SHA_CTX_fast SHA_CTX
    ++#    define platform_SHA1_Init_fast SHA1_Init
    ++#    define platform_SHA1_Update_fast SHA1_Update
    ++#    define platform_SHA1_Final_fast SHA1_Final
     +#  endif
    -+#  define platform_SHA_CTX_fast openssl_SHA1_CTX
    -+#  define platform_SHA1_Init_fast openssl_SHA1_Init
    -+#  define platform_SHA1_Clone_fast openssl_SHA1_Clone
    -+#  define platform_SHA1_Update_fast openssl_SHA1_Update
    -+#  define platform_SHA1_Final_fast openssl_SHA1_Final
     +#elif defined(SHA1_BLK_FAST)
    -+#include "block-sha1/sha1.h"
    -+#define platform_SHA_CTX_fast blk_SHA_CTX
    -+#define platform_SHA1_Init_fast blk_SHA1_Init
    -+#define platform_SHA1_Update_fast blk_SHA1_Update
    -+#define platform_SHA1_Final_fast blk_SHA1_Final
    ++#  include "block-sha1/sha1.h"
    ++#  define platform_SHA_CTX_fast blk_SHA_CTX
    ++#  define platform_SHA1_Init_fast blk_SHA1_Init
    ++#  define platform_SHA1_Update_fast blk_SHA1_Update
    ++#  define platform_SHA1_Final_fast blk_SHA1_Final
     +#endif
     +
      #if defined(SHA256_NETTLE)
4:  e8f5cbd280 = 4:  311fcc9596 csum-file.c: use fast SHA-1 implementation when available
--
2.46.0.426.g82754d92509.dirty

[PATCH v2 1/4] sha1: do not redefine `platform_SHA_CTX` and friends

[Taylor Blau]

Our in-tree SHA-1 wrappers all define platform_SHA_CTX and related
macros to point at the opaque "context" type, init, update, and similar
functions for each specific implementation.

In hash.h, we use these platform_ variables to set up the function
pointers for, e.g., the_hash_algo->init_fn(), etc.

But while these header files have a header-specific macro that prevents
them declaring their structs / functions multiple times, they
unconditionally define the platform variables, making it impossible to
load multiple SHA-1 implementations at once.

As a prerequisite for loading a separate SHA-1 implementation for
non-cryptographic uses, only define the platform_ variables if they have
not already been defined.

Signed-off-by: Taylor Blau <me@ttaylorr.com>
---
 block-sha1/sha1.h | 2 ++
 sha1/openssl.h    | 2 ++
 sha1dc_git.h      | 3 +++
 3 files changed, 7 insertions(+)

diff --git a/block-sha1/sha1.h b/block-sha1/sha1.h
index 9fb0441b988..47bb9166368 100644
--- a/block-sha1/sha1.h
+++ b/block-sha1/sha1.h
@@ -16,7 +16,9 @@ void blk_SHA1_Init(blk_SHA_CTX *ctx);
 void blk_SHA1_Update(blk_SHA_CTX *ctx, const void *dataIn, size_t len);
 void blk_SHA1_Final(unsigned char hashout[20], blk_SHA_CTX *ctx);
 
+#ifndef platform_SHA_CTX
 #define platform_SHA_CTX	blk_SHA_CTX
 #define platform_SHA1_Init	blk_SHA1_Init
 #define platform_SHA1_Update	blk_SHA1_Update
 #define platform_SHA1_Final	blk_SHA1_Final
+#endif
diff --git a/sha1/openssl.h b/sha1/openssl.h
index 006c1f4ba54..1038af47daf 100644
--- a/sha1/openssl.h
+++ b/sha1/openssl.h
@@ -40,10 +40,12 @@ static inline void openssl_SHA1_Clone(struct openssl_SHA1_CTX *dst,
 	EVP_MD_CTX_copy_ex(dst->ectx, src->ectx);
 }
 
+#ifndef platform_SHA_CTX
 #define platform_SHA_CTX openssl_SHA1_CTX
 #define platform_SHA1_Init openssl_SHA1_Init
 #define platform_SHA1_Clone openssl_SHA1_Clone
 #define platform_SHA1_Update openssl_SHA1_Update
 #define platform_SHA1_Final openssl_SHA1_Final
+#endif
 
 #endif /* SHA1_OPENSSL_H */
diff --git a/sha1dc_git.h b/sha1dc_git.h
index 60e3ce84395..f6f880cabea 100644
--- a/sha1dc_git.h
+++ b/sha1dc_git.h
@@ -18,7 +18,10 @@ void git_SHA1DCFinal(unsigned char [20], SHA1_CTX *);
 void git_SHA1DCUpdate(SHA1_CTX *ctx, const void *data, unsigned long len);
 
 #define platform_SHA_IS_SHA1DC /* used by "test-tool sha1-is-sha1dc" */
+
+#ifndef platform_SHA_CTX
 #define platform_SHA_CTX SHA1_CTX
 #define platform_SHA1_Init git_SHA1DCInit
 #define platform_SHA1_Update git_SHA1DCUpdate
 #define platform_SHA1_Final git_SHA1DCFinal
+#endif
-- 
2.46.0.426.g82754d92509.dirty

[PATCH v2 2/4] hash.h: scaffolding for _fast hashing variants

[Taylor Blau]

Git's default SHA-1 implementation is collision-detecting, which hardens
us against known SHA-1 attacks against Git objects. This makes Git
object writes safer at the expense of some speed when hashing through
the collision-detecting implementation, which is slower than
non-collision detecting alternatives.

Prepare for loading a separate "fast" SHA-1 implementation that can be
used for non-cryptographic purposes, like computing the checksum of
files that use the hashwrite() API.

This commit does not actually introduce any new compile-time knobs to
control which implementation is used as the fast SHA-1 variant, but does
add scaffolding so that the "git_hash_algo" structure has five new
function pointers which are "fast" variants of the five existing
hashing-related function pointers:

  - git_hash_init_fn fast_init_fn
  - git_hash_clone_fn fast_clone_fn
  - git_hash_update_fn fast_update_fn
  - git_hash_final_fn fast_final_fn
  - git_hash_final_oid_fn fast_final_oid_fn

The following commit will introduce compile-time knobs to specify which
SHA-1 implementation is used for non-cryptographic uses.

Signed-off-by: Taylor Blau <me@ttaylorr.com>
---
 hash.h        | 42 ++++++++++++++++++++++++++++++++++++++++++
 object-file.c | 42 ++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 84 insertions(+)

diff --git a/hash.h b/hash.h
index 72ffbc862e5..5e5b8205b58 100644
--- a/hash.h
+++ b/hash.h
@@ -44,14 +44,32 @@
 #define platform_SHA1_Final    	SHA1_Final
 #endif
 
+#ifndef platform_SHA_CTX_fast
+#  define platform_SHA_CTX_fast     platform_SHA_CTX
+#  define platform_SHA1_Init_fast   platform_SHA1_Init
+#  define platform_SHA1_Update_fast platform_SHA1_Update
+#  define platform_SHA1_Final_fast  platform_SHA1_Final
+#  ifdef platform_SHA1_Clone
+#    define platform_SHA1_Clone_fast platform_SHA1_Clone
+#  endif
+#endif
+
 #define git_SHA_CTX		platform_SHA_CTX
 #define git_SHA1_Init		platform_SHA1_Init
 #define git_SHA1_Update		platform_SHA1_Update
 #define git_SHA1_Final		platform_SHA1_Final
 
+#define git_SHA_CTX_fast	platform_SHA_CTX_fast
+#define git_SHA1_Init_fast	platform_SHA1_Init_fast
+#define git_SHA1_Update_fast	platform_SHA1_Update_fast
+#define git_SHA1_Final_fast	platform_SHA1_Final_fast
+
 #ifdef platform_SHA1_Clone
 #define git_SHA1_Clone	platform_SHA1_Clone
 #endif
+#ifdef platform_SHA1_Clone_fast
+#  define git_SHA1_Clone_fast platform_SHA1_Clone_fast
+#endif
 
 #ifndef platform_SHA256_CTX
 #define platform_SHA256_CTX	SHA256_CTX
@@ -81,6 +99,13 @@ static inline void git_SHA1_Clone(git_SHA_CTX *dst, const git_SHA_CTX *src)
 	memcpy(dst, src, sizeof(*dst));
 }
 #endif
+#ifndef SHA1_NEEDS_CLONE_HELPER_FAST
+static inline void git_SHA1_Clone_fast(git_SHA_CTX_fast *dst,
+				       const git_SHA_CTX_fast *src)
+{
+	memcpy(dst, src, sizeof(*dst));
+}
+#endif
 
 #ifndef SHA256_NEEDS_CLONE_HELPER
 static inline void git_SHA256_Clone(git_SHA256_CTX *dst, const git_SHA256_CTX *src)
@@ -178,6 +203,8 @@ enum get_oid_result {
 /* A suitably aligned type for stack allocations of hash contexts. */
 union git_hash_ctx {
 	git_SHA_CTX sha1;
+	git_SHA_CTX_fast sha1_fast;
+
 	git_SHA256_CTX sha256;
 };
 typedef union git_hash_ctx git_hash_ctx;
@@ -222,6 +249,21 @@ struct git_hash_algo {
 	/* The hash finalization function for object IDs. */
 	git_hash_final_oid_fn final_oid_fn;
 
+	/* The fast / non-cryptographic hash initialization function. */
+	git_hash_init_fn fast_init_fn;
+
+	/* The fast / non-cryptographic hash context cloning function. */
+	git_hash_clone_fn fast_clone_fn;
+
+	/* The fast / non-cryptographic hash update function. */
+	git_hash_update_fn fast_update_fn;
+
+	/* The fast / non-cryptographic hash finalization function. */
+	git_hash_final_fn fast_final_fn;
+
+	/* The fast / non-cryptographic hash finalization function. */
+	git_hash_final_oid_fn fast_final_oid_fn;
+
 	/* The OID of the empty tree. */
 	const struct object_id *empty_tree;
 
diff --git a/object-file.c b/object-file.c
index c5994202ba0..9691292ef5a 100644
--- a/object-file.c
+++ b/object-file.c
@@ -115,6 +115,33 @@ static void git_hash_sha1_final_oid(struct object_id *oid, git_hash_ctx *ctx)
 	oid->algo = GIT_HASH_SHA1;
 }
 
+static void git_hash_sha1_init_fast(git_hash_ctx *ctx)
+{
+	git_SHA1_Init_fast(&ctx->sha1_fast);
+}
+
+static void git_hash_sha1_clone_fast(git_hash_ctx *dst, const git_hash_ctx *src)
+{
+	git_SHA1_Clone_fast(&dst->sha1_fast, &src->sha1_fast);
+}
+
+static void git_hash_sha1_update_fast(git_hash_ctx *ctx, const void *data,
+				      size_t len)
+{
+	git_SHA1_Update_fast(&ctx->sha1_fast, data, len);
+}
+
+static void git_hash_sha1_final_fast(unsigned char *hash, git_hash_ctx *ctx)
+{
+	git_SHA1_Final_fast(hash, &ctx->sha1_fast);
+}
+
+static void git_hash_sha1_final_oid_fast(struct object_id *oid, git_hash_ctx *ctx)
+{
+	git_SHA1_Final_fast(oid->hash, &ctx->sha1_fast);
+	memset(oid->hash + GIT_SHA1_RAWSZ, 0, GIT_MAX_RAWSZ - GIT_SHA1_RAWSZ);
+	oid->algo = GIT_HASH_SHA1;
+}
 
 static void git_hash_sha256_init(git_hash_ctx *ctx)
 {
@@ -189,6 +216,11 @@ const struct git_hash_algo hash_algos[GIT_HASH_NALGOS] = {
 		.update_fn = git_hash_unknown_update,
 		.final_fn = git_hash_unknown_final,
 		.final_oid_fn = git_hash_unknown_final_oid,
+		.fast_init_fn = git_hash_unknown_init,
+		.fast_clone_fn = git_hash_unknown_clone,
+		.fast_update_fn = git_hash_unknown_update,
+		.fast_final_fn = git_hash_unknown_final,
+		.fast_final_oid_fn = git_hash_unknown_final_oid,
 		.empty_tree = NULL,
 		.empty_blob = NULL,
 		.null_oid = NULL,
@@ -204,6 +236,11 @@ const struct git_hash_algo hash_algos[GIT_HASH_NALGOS] = {
 		.update_fn = git_hash_sha1_update,
 		.final_fn = git_hash_sha1_final,
 		.final_oid_fn = git_hash_sha1_final_oid,
+		.fast_init_fn = git_hash_sha1_init_fast,
+		.fast_clone_fn = git_hash_sha1_clone_fast,
+		.fast_update_fn = git_hash_sha1_update_fast,
+		.fast_final_fn = git_hash_sha1_final_fast,
+		.fast_final_oid_fn = git_hash_sha1_final_oid_fast,
 		.empty_tree = &empty_tree_oid,
 		.empty_blob = &empty_blob_oid,
 		.null_oid = &null_oid_sha1,
@@ -219,6 +256,11 @@ const struct git_hash_algo hash_algos[GIT_HASH_NALGOS] = {
 		.update_fn = git_hash_sha256_update,
 		.final_fn = git_hash_sha256_final,
 		.final_oid_fn = git_hash_sha256_final_oid,
+		.fast_init_fn = git_hash_sha256_init,
+		.fast_clone_fn = git_hash_sha256_clone,
+		.fast_update_fn = git_hash_sha256_update,
+		.fast_final_fn = git_hash_sha256_final,
+		.fast_final_oid_fn = git_hash_sha256_final_oid,
 		.empty_tree = &empty_tree_oid_sha256,
 		.empty_blob = &empty_blob_oid_sha256,
 		.null_oid = &null_oid_sha256,
-- 
2.46.0.426.g82754d92509.dirty

[PATCH v2 3/4] Makefile: allow specifying a SHA-1 for non-cryptographic uses

[Taylor Blau]

Introduce _FAST variants of the OPENSSL_SHA1, BLK_SHA1, and
APPLE_COMMON_CRYPTO_SHA1 compile-time knobs which indicate which SHA-1
implementation is to be used for non-cryptographic uses.

There are a couple of small implementation notes worth mentioning:

  - There is no way to select the collision detecting SHA-1 as the
    "fast" fallback, since the fast fallback is only for
    non-cryptographic uses, and is meant to be faster than our
    collision-detecting implementation.

  - There are no similar knobs for SHA-256, since no collision attacks
    are presently known and thus no collision-detecting implementations
    actually exist.

Signed-off-by: Taylor Blau <me@ttaylorr.com>
---
 Makefile | 25 +++++++++++++++++++++++++
 hash.h   | 30 ++++++++++++++++++++++++++++++
 2 files changed, 55 insertions(+)

diff --git a/Makefile b/Makefile
index e298c8b55ec..d24f9088802 100644
--- a/Makefile
+++ b/Makefile
@@ -517,6 +517,10 @@ include shared.mak
 # Define APPLE_COMMON_CRYPTO_SHA1 to use Apple's CommonCrypto for
 # SHA-1.
 #
+# Define the same Makefile knobs as above, but suffixed with _FAST to
+# use the corresponding implementations for "fast" SHA-1 hashing for
+# non-cryptographic purposes.
+#
 # If don't enable any of the *_SHA1 settings in this section, Git will
 # default to its built-in sha1collisiondetection library, which is a
 # collision-detecting sha1 This is slower, but may detect attempted
@@ -1982,6 +1986,27 @@ endif
 endif
 endif
 
+ifdef OPENSSL_SHA1_FAST
+ifndef OPENSSL_SHA1
+	EXTLIBS += $(LIB_4_CRYPTO)
+	BASIC_CFLAGS += -DSHA1_OPENSSL_FAST
+endif
+else
+ifdef BLK_SHA1_FAST
+ifndef BLK_SHA1
+	LIB_OBJS += block-sha1/sha1.o
+	BASIC_CFLAGS += -DSHA1_BLK_FAST
+endif
+else
+ifdef APPLE_COMMON_CRYPTO_SHA1_FAST
+ifndef APPLE_COMMON_CRYPTO_SHA1
+	COMPAT_CFLAGS += -DCOMMON_DIGEST_FOR_OPENSSL
+	BASIC_CFLAGS += -DSHA1_APPLE_FAST
+endif
+endif
+endif
+endif
+
 ifdef OPENSSL_SHA256
 	EXTLIBS += $(LIB_4_CRYPTO)
 	BASIC_CFLAGS += -DSHA256_OPENSSL
diff --git a/hash.h b/hash.h
index 5e5b8205b58..6ea0d968017 100644
--- a/hash.h
+++ b/hash.h
@@ -15,6 +15,36 @@
 #include "block-sha1/sha1.h"
 #endif
 
+#if defined(SHA1_APPLE_FAST)
+#  include <CommonCrypto/CommonDigest.h>
+#  define platform_SHA_CTX_fast CC_SHA1_CTX
+#  define platform_SHA1_Init_fast CC_SHA1_Init
+#  define platform_SHA1_Update_fast CC_SHA1_Update
+#  define platform_SHA1_Final_fast CC_SHA1_Final
+#elif defined(SHA1_OPENSSL_FAST)
+#  include <openssl/sha.h>
+#  if defined(OPENSSL_API_LEVEL) && OPENSSL_API_LEVEL >= 3
+#    define SHA1_NEEDS_CLONE_HELPER_FAST
+#    include "sha1/openssl.h"
+#    define platform_SHA_CTX_fast openssl_SHA1_CTX
+#    define platform_SHA1_Init_fast openssl_SHA1_Init
+#    define platform_SHA1_Clone_fast openssl_SHA1_Clone
+#    define platform_SHA1_Update_fast openssl_SHA1_Update
+#    define platform_SHA1_Final_fast openssl_SHA1_Final
+#  else
+#    define platform_SHA_CTX_fast SHA_CTX
+#    define platform_SHA1_Init_fast SHA1_Init
+#    define platform_SHA1_Update_fast SHA1_Update
+#    define platform_SHA1_Final_fast SHA1_Final
+#  endif
+#elif defined(SHA1_BLK_FAST)
+#  include "block-sha1/sha1.h"
+#  define platform_SHA_CTX_fast blk_SHA_CTX
+#  define platform_SHA1_Init_fast blk_SHA1_Init
+#  define platform_SHA1_Update_fast blk_SHA1_Update
+#  define platform_SHA1_Final_fast blk_SHA1_Final
+#endif
+
 #if defined(SHA256_NETTLE)
 #include "sha256/nettle.h"
 #elif defined(SHA256_GCRYPT)
-- 
2.46.0.426.g82754d92509.dirty

[PATCH v2 4/4] csum-file.c: use fast SHA-1 implementation when available

[Taylor Blau]

Update hashwrite() and friends to use the fast_-variants of hashing
functions, calling for e.g., "the_hash_algo->fast_update_fn()" instead
of "the_hash_algo->update_fn()".

These callers only use the_hash_algo to produce a checksum, which we
depend on for data integrity, but not for cryptographic purposes, so
these callers are safe to use the fast (and potentially non-collision
detecting) SHA-1 implementation.

To time this, I took a freshly packed copy of linux.git, and ran the
following with and without the OPENSSL_SHA1_FAST=1 build-knob. Both
versions were compiled with -O3:

    $ git for-each-ref --format='%(objectname)' refs/heads refs/tags >in
    $ valgrind --tool=callgrind ~/src/git/git-pack-objects \
        --revs --stdout --all-progress --use-bitmap-index <in >/dev/null

Without OPENSSL_SHA1_FAST=1 (that is, using the collision-detecting
SHA-1 implementation for both cryptographic and non-cryptographic
purposes), we spend a significant amount of our instruction count in
hashwrite():

    $ callgrind_annotate --inclusive=yes | grep hashwrite | head -n1
    159,998,868,413 (79.42%)  /home/ttaylorr/src/git/csum-file.c:hashwrite [/home/ttaylorr/src/git/git-pack-objects]

, and the resulting "clone" takes 19.219 seconds of wall clock time,
18.94 seconds of user time and 0.28 seconds of system time.

Compiling with OPENSSL_SHA1_FAST=1, we spend ~60% fewer instructions in
hashwrite():

    $ callgrind_annotate --inclusive=yes | grep hashwrite | head -n1
     59,164,001,176 (58.79%)  /home/ttaylorr/src/git/csum-file.c:hashwrite [/home/ttaylorr/src/git/git-pack-objects]

, and generate the resulting "clone" much faster, in only 11.597 seconds
of wall time, 11.37 seconds of user time, and 0.23 seconds of system
time, for a ~40% speed-up.

Signed-off-by: Taylor Blau <me@ttaylorr.com>
---
 csum-file.c | 18 +++++++++---------
 1 file changed, 9 insertions(+), 9 deletions(-)

diff --git a/csum-file.c b/csum-file.c
index bf82ad8f9f5..cb8c39ecf3a 100644
--- a/csum-file.c
+++ b/csum-file.c
@@ -50,7 +50,7 @@ void hashflush(struct hashfile *f)
 
 	if (offset) {
 		if (!f->skip_hash)
-			the_hash_algo->update_fn(&f->ctx, f->buffer, offset);
+			the_hash_algo->fast_update_fn(&f->ctx, f->buffer, offset);
 		flush(f, f->buffer, offset);
 		f->offset = 0;
 	}
@@ -73,7 +73,7 @@ int finalize_hashfile(struct hashfile *f, unsigned char *result,
 	if (f->skip_hash)
 		hashclr(f->buffer, the_repository->hash_algo);
 	else
-		the_hash_algo->final_fn(f->buffer, &f->ctx);
+		the_hash_algo->fast_final_fn(f->buffer, &f->ctx);
 
 	if (result)
 		hashcpy(result, f->buffer, the_repository->hash_algo);
@@ -128,7 +128,7 @@ void hashwrite(struct hashfile *f, const void *buf, unsigned int count)
 			 * f->offset is necessarily zero.
 			 */
 			if (!f->skip_hash)
-				the_hash_algo->update_fn(&f->ctx, buf, nr);
+				the_hash_algo->fast_update_fn(&f->ctx, buf, nr);
 			flush(f, buf, nr);
 		} else {
 			/*
@@ -174,7 +174,7 @@ static struct hashfile *hashfd_internal(int fd, const char *name,
 	f->name = name;
 	f->do_crc = 0;
 	f->skip_hash = 0;
-	the_hash_algo->init_fn(&f->ctx);
+	the_hash_algo->fast_init_fn(&f->ctx);
 
 	f->buffer_len = buffer_len;
 	f->buffer = xmalloc(buffer_len);
@@ -208,7 +208,7 @@ void hashfile_checkpoint(struct hashfile *f, struct hashfile_checkpoint *checkpo
 {
 	hashflush(f);
 	checkpoint->offset = f->total;
-	the_hash_algo->clone_fn(&checkpoint->ctx, &f->ctx);
+	the_hash_algo->fast_clone_fn(&checkpoint->ctx, &f->ctx);
 }
 
 int hashfile_truncate(struct hashfile *f, struct hashfile_checkpoint *checkpoint)
@@ -219,7 +219,7 @@ int hashfile_truncate(struct hashfile *f, struct hashfile_checkpoint *checkpoint
 	    lseek(f->fd, offset, SEEK_SET) != offset)
 		return -1;
 	f->total = offset;
-	the_hash_algo->clone_fn(&f->ctx, &checkpoint->ctx);
+	the_hash_algo->fast_clone_fn(&f->ctx, &checkpoint->ctx);
 	f->offset = 0; /* hashflush() was called in checkpoint */
 	return 0;
 }
@@ -245,9 +245,9 @@ int hashfile_checksum_valid(const unsigned char *data, size_t total_len)
 	if (total_len < the_hash_algo->rawsz)
 		return 0; /* say "too short"? */
 
-	the_hash_algo->init_fn(&ctx);
-	the_hash_algo->update_fn(&ctx, data, data_len);
-	the_hash_algo->final_fn(got, &ctx);
+	the_hash_algo->fast_init_fn(&ctx);
+	the_hash_algo->fast_update_fn(&ctx, data, data_len);
+	the_hash_algo->fast_final_fn(got, &ctx);
 
 	return hasheq(got, data + data_len, the_repository->hash_algo);
 }
-- 
2.46.0.426.g82754d92509.dirty

[PATCH v3 0/9] hash.h: support choosing a separate SHA-1 for non-cryptographic uses

[Taylor Blau]

This series adds a build-time knob to allow selecting an alternative
SHA-1 implementation for non-cryptographic hashing within Git, starting
with the `hashwrite()` family of functions.

This version is a more size-able reroll from the first two rounds, which
updates pack-objects to use a (hardened) version of
`finalize_object_file()`, that is now sensitive to checksum collisions.

Peff and I wrote the first four (new) patches together, and I feel
confident now that we're in a good spot to address the concerns raised
in [1].

Thanks in advance for your review!

[1]: https://lore.kernel.org/git/20240905202707.GA2602440@coredump.intra.peff.net/

Taylor Blau (9):
  finalize_object_file(): check for name collision before renaming
  finalize_object_file(): refactor unlink_or_warn() placement
  finalize_object_file(): implement collision check
  pack-objects: use finalize_object_file() to rename pack/idx/etc
  i5500-git-daemon.sh: use compile-able version of Git without OpenSSL
  sha1: do not redefine `platform_SHA_CTX` and friends
  hash.h: scaffolding for _fast hashing variants
  Makefile: allow specifying a SHA-1 for non-cryptographic uses
  csum-file.c: use fast SHA-1 implementation when available

 Makefile                              |  25 ++++++
 block-sha1/sha1.h                     |   2 +
 csum-file.c                           |  18 ++---
 hash.h                                |  72 +++++++++++++++++
 object-file.c                         | 110 +++++++++++++++++++++++++-
 pack-write.c                          |   7 +-
 sha1/openssl.h                        |   2 +
 sha1dc_git.h                          |   3 +
 t/interop/i5500-git-daemon.sh         |   2 +-
 t/t5303-pack-corruption-resilience.sh |   7 +-
 10 files changed, 230 insertions(+), 18 deletions(-)

Range-diff against v2:
 -:  ----------- >  1:  738b1eb17b4 finalize_object_file(): check for name collision before renaming
 -:  ----------- >  2:  e1c2c39711f finalize_object_file(): refactor unlink_or_warn() placement
 -:  ----------- >  3:  0feee5d1d4f finalize_object_file(): implement collision check
 -:  ----------- >  4:  620dde48a9d pack-objects: use finalize_object_file() to rename pack/idx/etc
 -:  ----------- >  5:  bfe992765cd i5500-git-daemon.sh: use compile-able version of Git without OpenSSL
 1:  e7cd23bf4cd =  6:  22863d9f6df sha1: do not redefine `platform_SHA_CTX` and friends
 2:  3b5f21e4a62 =  7:  119c318d812 hash.h: scaffolding for _fast hashing variants
 3:  02764de1395 =  8:  137ec30d68a Makefile: allow specifying a SHA-1 for non-cryptographic uses
 4:  311fcc95960 =  9:  4018261366f csum-file.c: use fast SHA-1 implementation when available

base-commit: 159f2d50e75c17382c9f4eb7cbda671a6fa612d1
-- 
2.46.0.430.gca674632b70

[PATCH v3 1/9] finalize_object_file(): check for name collision before renaming

[Taylor Blau]

We prefer link()/unlink() to rename() for object files, with the idea
that we should prefer the data that is already on disk to what is
incoming. But we may fall back to rename() if the user has configured us
to do so, or if the filesystem seems not to support cross-directory
links. This loses the "prefer what is on disk" property.

We can mitigate this somewhat by trying to stat() the destination
filename before doing the rename. This is racy, since the object could
be created between the stat() and rename() calls. But in practice it is
expanding the definition of "what is already on disk" to be the point
that the function is called. That is enough to deal with any potential
attacks where an attacker is trying to collide hashes with what's
already in the repository.

Co-authored-by: Jeff King <peff@peff.net>
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Taylor Blau <me@ttaylorr.com>
---
 object-file.c | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/object-file.c b/object-file.c
index c5994202ba0..683e6b2a0bd 100644
--- a/object-file.c
+++ b/object-file.c
@@ -1904,6 +1904,7 @@ static void write_object_file_prepare_literally(const struct git_hash_algo *algo
  */
 int finalize_object_file(const char *tmpfile, const char *filename)
 {
+	struct stat st;
 	int ret = 0;
 
 	if (object_creation_mode == OBJECT_CREATION_USES_RENAMES)
@@ -1924,9 +1925,12 @@ int finalize_object_file(const char *tmpfile, const char *filename)
 	 */
 	if (ret && ret != EEXIST) {
 	try_rename:
-		if (!rename(tmpfile, filename))
+		if (!stat(filename, &st))
+			ret = EEXIST;
+		else if (!rename(tmpfile, filename))
 			goto out;
-		ret = errno;
+		else
+			ret = errno;
 	}
 	unlink_or_warn(tmpfile);
 	if (ret) {
-- 
2.46.0.430.gca674632b70

[PATCH v3 2/9] finalize_object_file(): refactor unlink_or_warn() placement

[Taylor Blau]

As soon as we've tried to link() a temporary object into place, we then
unlink() the tempfile immediately, whether we were successful or not.

For the success case, this is because we no longer need the old file
(it's now linked into place).

For the error case, there are two outcomes. Either we got EEXIST, in
which case we consider the collision to be a noop. Or we got a system
error, in which we case we are just cleaning up after ourselves.

Using a single line for all of these cases has some problems:

  - in the error case, our unlink() may clobber errno, which we use in
    the error message

  - for the collision case, there's a FIXME that indicates we should do
    a collision check. In preparation for implementing that, we'll need
    to actually hold on to the file.

Split these three cases into their own calls to unlink_or_warn(). This
is more verbose, but lets us do the right thing in each case.

Co-authored-by: Jeff King <peff@peff.net>
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Taylor Blau <me@ttaylorr.com>
---
 object-file.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/object-file.c b/object-file.c
index 683e6b2a0bd..54a82a5f7a0 100644
--- a/object-file.c
+++ b/object-file.c
@@ -1911,6 +1911,8 @@ int finalize_object_file(const char *tmpfile, const char *filename)
 		goto try_rename;
 	else if (link(tmpfile, filename))
 		ret = errno;
+	else
+		unlink_or_warn(tmpfile);
 
 	/*
 	 * Coda hack - coda doesn't like cross-directory links,
@@ -1932,12 +1934,15 @@ int finalize_object_file(const char *tmpfile, const char *filename)
 		else
 			ret = errno;
 	}
-	unlink_or_warn(tmpfile);
 	if (ret) {
 		if (ret != EEXIST) {
+			int saved_errno = errno;
+			unlink_or_warn(tmpfile);
+			errno = saved_errno;
 			return error_errno(_("unable to write file %s"), filename);
 		}
 		/* FIXME!!! Collision check here ? */
+		unlink_or_warn(tmpfile);
 	}
 
 out:
-- 
2.46.0.430.gca674632b70

[PATCH v3 3/9] finalize_object_file(): implement collision check

[Taylor Blau]

We've had "FIXME!!! Collision check here ?" in finalize_object_file()
since aac1794132 (Improve sha1 object file writing., 2005-05-03). That
is, when we try to write a file with the same name, we assume the
on-disk contents are the same and blindly throw away the new copy.

One of the reasons we never implemented this is because the files it
moves are all named after the cryptographic hash of their contents
(either loose objects, or packs which have their hash in the name these
days). So we are unlikely to see such a collision by accident. And even
though there are weaknesses in sha1, we assume they are mitigated by our
use of sha1dc.

So while it's a theoretical concern now, it hasn't been a priority.
However, if we start using weaker hashes for pack checksums and names,
this will become a practical concern. So in preparation, let's actually
implement a byte-for-byte collision check.

The new check will cause the write of new differing content to be a
failure, rather than a silent noop, and we'll retain the temporary file
on disk.

Note that this may cause some extra computation when the files are in
fact identical, but this should happen rarely. For example, when writing
a loose object, we compute the object id first, then check manually for
an existing object (so that we can freshen its timestamp) before
actually trying to write and link the new data.

Co-authored-by: Jeff King <peff@peff.net>
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Taylor Blau <me@ttaylorr.com>
---
 object-file.c | 55 +++++++++++++++++++++++++++++++++++++++++++++++++--
 1 file changed, 53 insertions(+), 2 deletions(-)

diff --git a/object-file.c b/object-file.c
index 54a82a5f7a0..85f91516429 100644
--- a/object-file.c
+++ b/object-file.c
@@ -1899,6 +1899,57 @@ static void write_object_file_prepare_literally(const struct git_hash_algo *algo
 	hash_object_body(algo, &c, buf, len, oid, hdr, hdrlen);
 }
 
+static int check_collision(const char *filename_a, const char *filename_b)
+{
+	char buf_a[4096], buf_b[4096];
+	int fd_a = -1, fd_b = -1;
+	int ret = 0;
+
+	fd_a = open(filename_a, O_RDONLY);
+	if (fd_a < 0) {
+		ret = error_errno(_("unable to open %s"), filename_a);
+		goto out;
+	}
+
+	fd_b = open(filename_b, O_RDONLY);
+	if (fd_b < 0) {
+		ret = error_errno(_("unable to open %s"), filename_b);
+		goto out;
+	}
+
+	while (1) {
+		ssize_t sz_a, sz_b;
+
+		sz_a = read_in_full(fd_a, buf_a, sizeof(buf_a));
+		if (sz_a < 0) {
+			ret = error_errno(_("unable to read %s"), filename_a);
+			goto out;
+		}
+
+		sz_b = read_in_full(fd_b, buf_b, sizeof(buf_b));
+		if (sz_b < 0) {
+			ret = error_errno(_("unable to read %s"), filename_b);
+			goto out;
+		}
+
+		if (sz_a != sz_b || memcmp(buf_a, buf_b, sz_a)) {
+			ret = error(_("files '%s' and '%s' differ in contents"),
+				    filename_a, filename_b);
+			goto out;
+		}
+
+		if (sz_a < sizeof(buf_a))
+			break;
+	}
+
+out:
+	if (fd_a > -1)
+		close(fd_a);
+	if (fd_b > -1)
+		close(fd_b);
+	return ret;
+}
+
 /*
  * Move the just written object into its final resting place.
  */
@@ -1941,8 +1992,8 @@ int finalize_object_file(const char *tmpfile, const char *filename)
 			errno = saved_errno;
 			return error_errno(_("unable to write file %s"), filename);
 		}
-		/* FIXME!!! Collision check here ? */
-		unlink_or_warn(tmpfile);
+		if (check_collision(tmpfile, filename))
+			return -1;
 	}
 
 out:
-- 
2.46.0.430.gca674632b70

Re: [PATCH v3 3/9] finalize_object_file(): implement collision check

[Junio C Hamano]

Taylor Blau <me@ttaylorr.com> writes:

> Note that this may cause some extra computation when the files are in
> fact identical, but this should happen rarely. For example, when writing
> a loose object, we compute the object id first, then check manually for
> an existing object (so that we can freshen its timestamp) before
> actually trying to write and link the new data.

True.

> +static int check_collision(const char *filename_a, const char *filename_b)
> +{
> +	char buf_a[4096], buf_b[4096];
> +	int fd_a = -1, fd_b = -1;
> +	int ret = 0;
> +
> +	fd_a = open(filename_a, O_RDONLY);
> +	if (fd_a < 0) {
> +		ret = error_errno(_("unable to open %s"), filename_a);
> +		goto out;
> +	}
> +
> +	fd_b = open(filename_b, O_RDONLY);
> +	if (fd_b < 0) {
> +		ret = error_errno(_("unable to open %s"), filename_b);
> +		goto out;
> +	}

Two and two half comments on this function.

 * We compare 4k at a time here, while copy.c copies 8k at a time,
   and bulk-checkin.c uses 16k at a time.  Outside the scope of this
   topic, we probably should pick one number and stick to it, unless
   we have measured to pick perfect number for each case (and I know
   I picked 8k for copy.c and 16k for bulk-checkin.c both out of
   thin air).

 * I would have expected at least we would fstat() them to declare
   difference immediately after we find their sizes differ, for
   example.  As we assume that calling into this function should be
   rare, we prefer not to pay in complexity for performance here?

 * We use read_in_full() and assume that a short-read return from
   the function happens only at the end of file due to EOF, which is
   another reason why we can do away without fstat() on these files.

 * An error causes the caller to assume collision (because we assign
   the return value of error() to ret), which should do the same
   action as an actual collision to abort and keep the problematic
   file for forensics.

>  /*
>   * Move the just written object into its final resting place.
>   */
> @@ -1941,8 +1992,8 @@ int finalize_object_file(const char *tmpfile, const char *filename)
>  			errno = saved_errno;
>  			return error_errno(_("unable to write file %s"), filename);
>  		}
> -		/* FIXME!!! Collision check here ? */
> -		unlink_or_warn(tmpfile);
> +		if (check_collision(tmpfile, filename))
> +			return -1;
>  	}

OK.

Re: [PATCH v3 3/9] finalize_object_file(): implement collision check

[Chris Torek]

On Fri, Sep 6, 2024 at 2:44 PM Junio C Hamano <gitster@pobox.com> wrote:
> Two and two half comments on [check_collision].
>
>  * We compare 4k at a time here, while copy.c copies 8k at a time,
>    and bulk-checkin.c uses 16k at a time.  Outside the scope of this
>    topic, we probably should pick one number and stick to it, unless
>    we have measured to pick perfect number for each case (and I know
>    I picked 8k for copy.c and 16k for bulk-checkin.c both out of
>    thin air).

In Ye Olden Days, 4k would be fine, and going back 40+ years
even 512 would be fine. For *writing* modern systems often
prefer at least 128K or even 1M; anything under 8K is Right Out.
For *writing* it tends to be less important due to caches.  Still:

>  * I would have expected at least we would fstat() them to declare
>    difference immediately after we find their sizes differ, for
>    example.  As we assume that calling into this function should be
>    rare, we prefer not to pay in complexity for performance here?

Another benefit of calling `stat` is that you get `st_blksize`, which
is the system's recommended I/O block size.

I almost commented about this earlier but the "should be rare"
thing held me back. :-)

(I have no comments on the rest of the comments.)

Chris

Re: [PATCH v3 3/9] finalize_object_file(): implement collision check

[Jeff King]

On Fri, Sep 06, 2024 at 02:44:12PM -0700, Junio C Hamano wrote:

> > +static int check_collision(const char *filename_a, const char *filename_b)
> [...]
> 
> Two and two half comments on this function.
> 
>  * We compare 4k at a time here, while copy.c copies 8k at a time,
>    and bulk-checkin.c uses 16k at a time.  Outside the scope of this
>    topic, we probably should pick one number and stick to it, unless
>    we have measured to pick perfect number for each case (and I know
>    I picked 8k for copy.c and 16k for bulk-checkin.c both out of
>    thin air).

I can confirm that 4k was picked out of thin air by me while writing
this function. ;) It's my usual size for I/O just because it often
matches the page size. I wouldn't be surprised if other values are
faster, but I hope that this code would run pretty infrequently.

So I wouldn't worry too much about it, but if there's an obviously
better I/O size and we have a #define for it, it would make sense to use
it. I do wonder if we'd run into stack limitations at some point.

As an alternative, we could also mmap the files and then compare the
full arrays directly. That might be faster?

>  * I would have expected at least we would fstat() them to declare
>    difference immediately after we find their sizes differ, for
>    example.  As we assume that calling into this function should be
>    rare, we prefer not to pay in complexity for performance here?

I actually had the opposite thought about fstat() performance while
writing it.  In a world where you expect most name collisions to
actually have different content, then checking their sizes lets you skip
the more expensive byte-wise check. But in a world where you mostly
expect them _not_ to differ, then the size check usually does not help,
and you waste time doing it. I.e., it is a cache-miss problem with
weighted costs.

Now thinking on it more, that view is probably dumb. fstat() is really
cheap, and byte-wise comparisons are really expensive, so if it has even
a tiny chance of helping, it might be worth doing.

Though again, I'd hope this will trigger pretty rarely in practice,
because it's probably a sign that we could have skipped work earlier
(i.e., by realizing we were just going to generate an identical file,
and not generated it in the first place).

So I'd be content with just about any implementation, and waiting to see
if it ever becomes a performance pain point.

>  * We use read_in_full() and assume that a short-read return from
>    the function happens only at the end of file due to EOF, which is
>    another reason why we can do away without fstat() on these files.
> 
>  * An error causes the caller to assume collision (because we assign
>    the return value of error() to ret), which should do the same
>    action as an actual collision to abort and keep the problematic
>    file for forensics.

Yup, both of those were very intentional. :)

-Peff

Re: [PATCH v3 3/9] finalize_object_file(): implement collision check

[Junio C Hamano]

Jeff King <peff@peff.net> writes:

> Now thinking on it more, that view is probably dumb. fstat() is really
> cheap, and byte-wise comparisons are really expensive, so if it has even
> a tiny chance of helping, it might be worth doing.

If we see too many "yikes we need to check for collision" cases,
then yes, but I agree that it should be rare to matter.

> Though again, I'd hope this will trigger pretty rarely in practice,
> because it's probably a sign that we could have skipped work earlier
> (i.e., by realizing we were just going to generate an identical file,
> and not generated it in the first place).

Yes, exactly.

Re: [PATCH v3 3/9] finalize_object_file(): implement collision check

[Patrick Steinhardt]

On Fri, Sep 06, 2024 at 03:46:12PM -0400, Taylor Blau wrote:
> diff --git a/object-file.c b/object-file.c
> index 54a82a5f7a0..85f91516429 100644
> --- a/object-file.c
> +++ b/object-file.c
> @@ -1899,6 +1899,57 @@ static void write_object_file_prepare_literally(const struct git_hash_algo *algo
>  	hash_object_body(algo, &c, buf, len, oid, hdr, hdrlen);
>  }
>  
> +static int check_collision(const char *filename_a, const char *filename_b)
> +{
> +	char buf_a[4096], buf_b[4096];
> +	int fd_a = -1, fd_b = -1;
> +	int ret = 0;
> +
> +	fd_a = open(filename_a, O_RDONLY);
> +	if (fd_a < 0) {
> +		ret = error_errno(_("unable to open %s"), filename_a);
> +		goto out;
> +	}
> +
> +	fd_b = open(filename_b, O_RDONLY);
> +	if (fd_b < 0) {
> +		ret = error_errno(_("unable to open %s"), filename_b);
> +		goto out;
> +	}
> +
> +	while (1) {
> +		ssize_t sz_a, sz_b;
> +
> +		sz_a = read_in_full(fd_a, buf_a, sizeof(buf_a));
> +		if (sz_a < 0) {
> +			ret = error_errno(_("unable to read %s"), filename_a);
> +			goto out;
> +		}
> +
> +		sz_b = read_in_full(fd_b, buf_b, sizeof(buf_b));
> +		if (sz_b < 0) {
> +			ret = error_errno(_("unable to read %s"), filename_b);
> +			goto out;
> +		}
> +
> +		if (sz_a != sz_b || memcmp(buf_a, buf_b, sz_a)) {
> +			ret = error(_("files '%s' and '%s' differ in contents"),
> +				    filename_a, filename_b);
> +			goto out;
> +		}
> +
> +		if (sz_a < sizeof(buf_a))
> +			break;
> +	}
> +
> +out:
> +	if (fd_a > -1)
> +		close(fd_a);
> +	if (fd_b > -1)
> +		close(fd_b);
> +	return ret;
> +}
> +
>  /*
>   * Move the just written object into its final resting place.
>   */

This function compares the exact contents, but isn't that wrong? The
contents may differ even though the object is the same because the
object hash is derived from the uncompressed data, whereas we store
compressed data on disk.

So this might trigger when you have different zlib versions, but also if
you configure core.compression differently.

Patrick

Re: [PATCH v3 3/9] finalize_object_file(): implement collision check

[Taylor Blau]

On Mon, Sep 16, 2024 at 12:45:38PM +0200, Patrick Steinhardt wrote:
> This function compares the exact contents, but isn't that wrong? The
> contents may differ even though the object is the same because the
> object hash is derived from the uncompressed data, whereas we store
> compressed data on disk.
>
> So this might trigger when you have different zlib versions, but also if
> you configure core.compression differently.

Oh, shoot -- you're right for when we call finalize_object_file() on
loose objects.

For packfiles and other spots where we use the hashfile API, the
name of the file depends on the checksum'd contents, so we're safe
there. But for loose objects the, the name of the file is based on the
object hash, which is the hash of the uncompressed contents.

So I think we would need something like this on top:

--- 8< ---
diff --git a/object-file.c b/object-file.c
index 84dd7d0fab..2f1616651e 100644
--- a/object-file.c
+++ b/object-file.c
@@ -1992,10 +1992,8 @@ static int check_collision(const char *filename_a, const char *filename_b)
 	return ret;
 }

-/*
- * Move the just written object into its final resting place.
- */
-int finalize_object_file(const char *tmpfile, const char *filename)
+static int finalize_object_file_1(const char *tmpfile, const char *filename,
+				  unsigned check_collisions)
 {
 	struct stat st;
 	int ret = 0;
@@ -2044,6 +2042,14 @@ int finalize_object_file(const char *tmpfile, const char *filename)
 	return 0;
 }

+/*
+ * Move the just written object into its final resting place.
+ */
+int finalize_object_file(const char *tmpfile, const char *filename)
+{
+	return finalize_object_file_1(tmpfile, filename, 1);
+}
+
 static void hash_object_file_literally(const struct git_hash_algo *algo,
 				       const void *buf, unsigned long len,
 				       const char *type, struct object_id *oid)
@@ -2288,7 +2294,7 @@ static int write_loose_object(const struct object_id *oid, char *hdr,
 			warning_errno(_("failed utime() on %s"), tmp_file.buf);
 	}

-	return finalize_object_file(tmp_file.buf, filename.buf);
+	return finalize_object_file_1(tmp_file.buf, filename.buf, 0);
 }

 static int freshen_loose_object(const struct object_id *oid)
@@ -2410,7 +2416,7 @@ int stream_loose_object(struct input_stream *in_stream, size_t len,
 		strbuf_release(&dir);
 	}

-	err = finalize_object_file(tmp_file.buf, filename.buf);
+	err = finalize_object_file_1(tmp_file.buf, filename.buf, 0);
 	if (!err && compat)
 		err = repo_add_loose_object_map(the_repository, oid, &compat_oid);
 cleanup:
--- >8 ---

But I'd like for some others to take a look at this before I send a new
round that includes this change.

Thanks for catching that!

Thanks,
Taylor

Re: [PATCH v3 3/9] finalize_object_file(): implement collision check

[Taylor Blau]

On Mon, Sep 16, 2024 at 11:54:14AM -0400, Taylor Blau wrote:
> On Mon, Sep 16, 2024 at 12:45:38PM +0200, Patrick Steinhardt wrote:
> > This function compares the exact contents, but isn't that wrong? The
> > contents may differ even though the object is the same because the
> > object hash is derived from the uncompressed data, whereas we store
> > compressed data on disk.
> >
> > So this might trigger when you have different zlib versions, but also if
> > you configure core.compression differently.
>
> Oh, shoot -- you're right for when we call finalize_object_file() on
> loose objects.
>
> For packfiles and other spots where we use the hashfile API, the
> name of the file depends on the checksum'd contents, so we're safe
> there. But for loose objects the, the name of the file is based on the
> object hash, which is the hash of the uncompressed contents.
>
> So I think we would need something like this on top:

Thinking about this a little more, I think that most cases should
actually be OK. Of course, this only affects repositories that are
changing their zlib version, and/or changing their core.compression
setting regularly, so this is all pretty niche, but still...

We often guard calls to write_loose_object() with either calling
freshen_loose_object(), or freshen_packed_object(), which will update
the mtime of the containing pack or loose object path for a given hash.

So if we already have a loose object with hash X in the object store,
and we try to write that same object again with a different zlib
version and/or core.compression setting, we'll simply call
freshen_loose_object() and optimize out the actual object write.

Of course, that's not always the case. We might e.g., force an object
loose via loosen_packed_objects() which could encounter a TOCTOU race
with an incoming object write using a different compression setting. But
that seems exceedingly rare to me.

I think that we should still take that change in the mail I'm replying
to, but I would definitely appreciate others' thoughts here.

Thanks,
Taylor

Re: [PATCH v3 3/9] finalize_object_file(): implement collision check

[Junio C Hamano]

Patrick Steinhardt <ps@pks.im> writes:

>> +static int check_collision(const char *filename_a, const char *filename_b)
>> +{
>> ...
>> +	return ret;
>> +}
>> +
>>  /*
>>   * Move the just written object into its final resting place.
>>   */
>
> This function compares the exact contents, but isn't that wrong? The
> contents may differ even though the object is the same because the
> object hash is derived from the uncompressed data, whereas we store
> compressed data on disk.

Very true.  So check_collision is *not* a good name for this helper
function.  

For .pack files, a collision means two resulting files have the
identical contents, so the above function, perhaps renamed to
"compare_files()" or something, is a very appropriate helper to
detect a collision once you get two files that claim to be the same
.pack file.

For loose object files, a collision means the contents before
compression are the same, so if we wanted to check collisions within
the same framework, we'd need "compare_files_after_inflating()" or
something, that opens both files and compare their contents while
"inflating" via zlib.

Thanks.

[PATCH v3 4/9] pack-objects: use finalize_object_file() to rename pack/idx/etc

[Taylor Blau]

In most places that write files to the object database (even packfiles
via index-pack or fast-import), we use finalize_object_file(). This
prefers link()/unlink() over rename(), because it means we will prefer
data that is already in the repository to data that we are newly
writing.

We should do the same thing in pack-objects. Even though we don't think
of it as accepting outside data (and thus not being susceptible to
collision attacks), in theory a determined attacker could present just
the right set of objects to cause an incremental repack to generate
a pack with their desired hash.

This has some test and real-world fallout, as seen in the adjustment to
t5303 below. That test script assumes that we can "fix" corruption by
repacking into a good state, including when the pack generated by that
repack operation collides with a (corrupted) pack with the same hash.
This violates our assumption from the previous adjustments to
finalize_object_file() that if we're moving a new file over an existing
one, that since their checksums match, so too must their contents.

This makes "fixing" corruption like this a more explicit operation,
since the test (and users, who may fix real-life corruption using a
similar technique) must first move the broken contents out of the way.

Note also that we now call adjust_shared_perm() twice. We already call
adjust_shared_perm() in stage_tmp_packfiles(), and now call it again in
finalize_object_file(). This is somewhat wasteful, but cleaning up the
existing calls to adjust_shared_perm() is tricky (because sometimes
we're writing to a tmpfile, and sometimes we're writing directly into
the final destination), so let's tolerate some minor waste until we can
more carefully clean up the now-redundant calls.

Co-authored-by: Jeff King <peff@peff.net>
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Taylor Blau <me@ttaylorr.com>
---
 pack-write.c                          | 7 ++++---
 t/t5303-pack-corruption-resilience.sh | 7 ++++++-
 2 files changed, 10 insertions(+), 4 deletions(-)

diff --git a/pack-write.c b/pack-write.c
index d07f03d0ab0..e5beecd3a4f 100644
--- a/pack-write.c
+++ b/pack-write.c
@@ -8,6 +8,7 @@
 #include "csum-file.h"
 #include "remote.h"
 #include "chunk-format.h"
+#include "object-file.h"
 #include "pack-mtimes.h"
 #include "pack-objects.h"
 #include "pack-revindex.h"
@@ -527,9 +528,9 @@ static void rename_tmp_packfile(struct strbuf *name_prefix, const char *source,
 	size_t name_prefix_len = name_prefix->len;
 
 	strbuf_addstr(name_prefix, ext);
-	if (rename(source, name_prefix->buf))
-		die_errno("unable to rename temporary file to '%s'",
-			  name_prefix->buf);
+	if (finalize_object_file(source, name_prefix->buf))
+		die("unable to rename temporary file to '%s'",
+		    name_prefix->buf);
 	strbuf_setlen(name_prefix, name_prefix_len);
 }
 
diff --git a/t/t5303-pack-corruption-resilience.sh b/t/t5303-pack-corruption-resilience.sh
index 61469ef4a68..e6a43ec9ae3 100755
--- a/t/t5303-pack-corruption-resilience.sh
+++ b/t/t5303-pack-corruption-resilience.sh
@@ -44,9 +44,14 @@ create_new_pack() {
 }
 
 do_repack() {
+    for f in $pack.*
+    do
+	    mv $f "$(echo $f | sed -e 's/pack-/pack-corrupt-/')" || return 1
+    done &&
     pack=$(printf "$blob_1\n$blob_2\n$blob_3\n" |
           git pack-objects $@ .git/objects/pack/pack) &&
-    pack=".git/objects/pack/pack-${pack}"
+    pack=".git/objects/pack/pack-${pack}" &&
+    rm -f .git/objects/pack/pack-corrupt-*
 }
 
 do_corrupt_object() {
-- 
2.46.0.430.gca674632b70

[PATCH v3 5/9] i5500-git-daemon.sh: use compile-able version of Git without OpenSSL

[Taylor Blau]

For the interop tests exercising basic 'git daemon' functionality, we
use version v1.0.0 as the old version of Git (which in this test we
happen to designate with using VERSION_B).

But that version does not compile with modern versions of OpenSSL,
complaining with error messages like:

    epoch.c:21:16: error: field ‘numerator’ has incomplete type
       21 |         BIGNUM numerator;
          |                ^~~~~~~~~
    epoch.c:22:16: error: field ‘denominator’ has incomplete type
       22 |         BIGNUM denominator;
          |                ^~~~~~~~~~~
    epoch.c: In function ‘new_zero’:

Of course, compiling with `NO_OPENSSL=1`, which we have had since
dd53c7ab297 ([PATCH] Support for NO_OPENSSL, 2005-07-29) allows us to
compile cleanly.

This hasn't been such a problem in practice because most builds can use
NO_OPENSSL when compiling the older versions of Git used by the interop
tests, because often even the current version of Git does not use
OpenSSL (e.g., because we use the collision detecting implementation of
SHA-1).

But subsequent changes will make a build configuration that does use
OpenSSL's SHA-1 implementation (at least for non-cryptographic uses)
more common, thus breaking this interop build (since only one side will
compile with NO_OPENSSL).

Let's work around the issue by using a slightly more modern, but still
quite old v1.6.6.3, which is used by the i0000-basic.sh test script as
well.

Signed-off-by: Taylor Blau <me@ttaylorr.com>
---
 t/interop/i5500-git-daemon.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/t/interop/i5500-git-daemon.sh b/t/interop/i5500-git-daemon.sh
index 4d22e42f842..c5bf37e4739 100755
--- a/t/interop/i5500-git-daemon.sh
+++ b/t/interop/i5500-git-daemon.sh
@@ -1,7 +1,7 @@
 #!/bin/sh
 
 VERSION_A=.
-VERSION_B=v1.0.0
+VERSION_B=v1.6.6.3
 
 : ${LIB_GIT_DAEMON_PORT:=5500}
 LIB_GIT_DAEMON_COMMAND='git.a daemon'
-- 
2.46.0.430.gca674632b70

Re: [PATCH v3 5/9] i5500-git-daemon.sh: use compile-able version of Git without OpenSSL

[Jeff King]

On Fri, Sep 06, 2024 at 03:46:19PM -0400, Taylor Blau wrote:

> Let's work around the issue by using a slightly more modern, but still
> quite old v1.6.6.3, which is used by the i0000-basic.sh test script as
> well.

So I know I shouldn't care that much about which ancient version the
interop tests are using. But it feels like we should be able to provide
the tools to make this work.

How about this instead?

-- >8 --
Subject: [PATCH] t/interop: allow per-version make options

Building older versions of Git may require tweaking some build knobs. In
particular, very old versions of Git will fail to build with recent
OpenSSL, because the bignum type switched from a struct to a pointer.

The i5500 interop test uses Git v1.0.0 by default, which triggers this
problem. You can work around it by setting NO_OPENSSL in your
GIT_TEST_MAKE_OPTS variable. But there are two downsides:

  1. You have to know to do this, and it's not at all obvious.

  2. That sets the options for _all_ versions of Git that we build. And
     it's possible for two versions to require conflicting knobs. E.g.,
     building with "make NO_OPENSSL=Nope OPENSSL_SHA1=Yes" causes
     imap-send.c to barf, because it declares a fallback typdef for SSL.
     This is something we may want to fix, but of course many historical
     versions are affected, and the interop scripts should be flexible
     enough to build everything.

So let's introduce per-version make options, along with the ability for
scripts to specify knobs that match their default versions. That should
make everything build out of the box, but also allow testers flexibility
if they are testing interoperability between non-default versions.

We'll set NO_OPENSSL by default for v1.0.0 in i5500. It doesn't have to
worry about the conflict with OPENSSL_SHA1 because imap-send did not
exist back then (but if it did, it could also just explicitly use a
different hash implementation).

Signed-off-by: Jeff King <peff@peff.net>
---
 t/interop/README              | 7 +++++++
 t/interop/i5500-git-daemon.sh | 1 +
 t/interop/interop-lib.sh      | 8 +++++---
 3 files changed, 13 insertions(+), 3 deletions(-)

diff --git a/t/interop/README b/t/interop/README
index 72d42bd856..4e0608f857 100644
--- a/t/interop/README
+++ b/t/interop/README
@@ -83,3 +83,10 @@ You can then use test_expect_success as usual, with a few differences:
      should create one with the appropriate version of git.
 
 At the end of the script, call test_done as usual.
+
+Some older versions may need a few build knobs tweaked (e.g., ancient
+versions of Git no longer build with modern OpenSSL). Your script can
+set MAKE_OPTS_A and MAKE_OPTS_B, which will be passed alongside
+GIT_INTEROP_MAKE_OPTS. Users can override them per-script by setting
+GIT_INTEROP_MAKE_OPTS_{A,B} in the environment, just like they can set
+GIT_TEST_VERSION_{A,B}.
diff --git a/t/interop/i5500-git-daemon.sh b/t/interop/i5500-git-daemon.sh
index 4d22e42f84..88712d1b5d 100755
--- a/t/interop/i5500-git-daemon.sh
+++ b/t/interop/i5500-git-daemon.sh
@@ -2,6 +2,7 @@
 
 VERSION_A=.
 VERSION_B=v1.0.0
+MAKE_OPTS_B="NO_OPENSSL=TooOld"
 
 : ${LIB_GIT_DAEMON_PORT:=5500}
 LIB_GIT_DAEMON_COMMAND='git.a daemon'
diff --git a/t/interop/interop-lib.sh b/t/interop/interop-lib.sh
index 62f4481b6e..1b5864d2a7 100644
--- a/t/interop/interop-lib.sh
+++ b/t/interop/interop-lib.sh
@@ -45,7 +45,7 @@ build_version () {
 
 	(
 		cd "$dir" &&
-		make $GIT_INTEROP_MAKE_OPTS >&2 &&
+		make $2 $GIT_INTEROP_MAKE_OPTS >&2 &&
 		touch .built
 	) || return 1
 
@@ -76,9 +76,11 @@ generate_wrappers () {
 
 VERSION_A=${GIT_TEST_VERSION_A:-$VERSION_A}
 VERSION_B=${GIT_TEST_VERSION_B:-$VERSION_B}
+MAKE_OPTS_A=${GIT_INTEROP_MAKE_OPTS_A:-$MAKE_OPTS_A}
+MAKE_OPTS_B=${GIT_INTEROP_MAKE_OPTS_B:-$MAKE_OPTS_B}
 
-if ! DIR_A=$(build_version "$VERSION_A") ||
-   ! DIR_B=$(build_version "$VERSION_B")
+if ! DIR_A=$(build_version "$VERSION_A" "$MAKE_OPTS_A") ||
+   ! DIR_B=$(build_version "$VERSION_B" "$MAKE_OPTS_B")
 then
 	echo >&2 "fatal: unable to build git versions"
 	exit 1
-- 
2.46.0.883.g5805d96482

Re: [PATCH v3 5/9] i5500-git-daemon.sh: use compile-able version of Git without OpenSSL

[Jeff King]

On Wed, Sep 11, 2024 at 02:10:10AM -0400, Jeff King wrote:

>   2. That sets the options for _all_ versions of Git that we build. And
>      it's possible for two versions to require conflicting knobs. E.g.,
>      building with "make NO_OPENSSL=Nope OPENSSL_SHA1=Yes" causes
>      imap-send.c to barf, because it declares a fallback typdef for SSL.
>      This is something we may want to fix, but of course many historical
>      versions are affected, and the interop scripts should be flexible
>      enough to build everything.

And here's the fix to make this combo work (and likewise, the "fast"
variant). We'd still want the interop fix for the reasons given above,
but it feels like one less gotcha for people to hit if they are using
OPENSSL_SHA1_FAST.

-- >8 --
Subject: [PATCH] imap-send: handle NO_OPENSSL even when openssl exists

If NO_OPENSSL is defined, then imap-send.c defines a fallback "SSL"
type, which is just a void pointer that remains NULL. This works, but it
has one problem: it is using the type name "SSL", which conflicts with
the upstream name, if some other part of the system happens to include
openssl. For example:

  $ make NO_OPENSSL=Nope OPENSSL_SHA1=Yes imap-send.o
      CC imap-send.o
  imap-send.c:35:15: error: conflicting types for ‘SSL’; have ‘void *’
     35 | typedef void *SSL;
        |               ^~~
  In file included from /usr/include/openssl/evp.h:26,
                   from sha1/openssl.h:4,
                   from hash.h:10,
                   from object.h:4,
                   from commit.h:4,
                   from refs.h:4,
                   from setup.h:4,
                   from imap-send.c:32:
  /usr/include/openssl/types.h:187:23: note: previous declaration of ‘SSL’ with type ‘SSL’ {aka ‘struct ssl_st’}
    187 | typedef struct ssl_st SSL;
        |                       ^~~
  make: *** [Makefile:2761: imap-send.o] Error 1

This is not a terribly common combination in practice:

  1. Why are we disabling openssl support but still using its sha1? The
     answer is that you may use the same build options across many
     versions, and some older versions of Git no longer build with
     modern versions of openssl.

  2. Why are we using a totally unsafe sha1 that does not detect
     collisions? You're right, we shouldn't. But in preparation for
     using unsafe sha1 for non-cryptographic checksums, it would be nice
     to be able to turn it on without hassle.

We can make this work by adjusting the way imap-send handles its
fallback. One solution is something like this:

  #ifdef NO_OPENSSL
  #define git_SSL void *
  #else
  #define git_SSL SSL
  #endif

But we can observe that we only need this definition in one spot: the
struct which holds the variable. So rather than play around with macros
that may cause unexpected effects, we can just directly use the correct
type in that struct.

Signed-off-by: Jeff King <peff@peff.net>
---
 imap-send.c | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/imap-send.c b/imap-send.c
index 2dd42807cd..ec68a06687 100644
--- a/imap-send.c
+++ b/imap-send.c
@@ -31,9 +31,6 @@
 #include "parse-options.h"
 #include "setup.h"
 #include "strbuf.h"
-#if defined(NO_OPENSSL) && !defined(HAVE_OPENSSL_CSPRNG)
-typedef void *SSL;
-#endif
 #ifdef USE_CURL_FOR_IMAP_SEND
 #include "http.h"
 #endif
@@ -85,7 +82,11 @@ struct imap_server_conf {
 
 struct imap_socket {
 	int fd[2];
+#if defined(NO_OPENSSL) && !defined(HAVE_OPENSSL_CSPRNG)
+	void *ssl;
+#else
 	SSL *ssl;
+#endif
 };
 
 struct imap_buffer {
-- 
2.46.0.883.g5805d96482

Re: [PATCH v3 5/9] i5500-git-daemon.sh: use compile-able version of Git without OpenSSL

[Junio C Hamano]

Jeff King <peff@peff.net> writes:

> Subject: [PATCH] imap-send: handle NO_OPENSSL even when openssl exists
> ...
> But we can observe that we only need this definition in one spot: the
> struct which holds the variable. So rather than play around with macros
> that may cause unexpected effects, we can just directly use the correct
> type in that struct.
>
> Signed-off-by: Jeff King <peff@peff.net>
> ---
>  imap-send.c | 7 ++++---
>  1 file changed, 4 insertions(+), 3 deletions(-)

Neat.  Will queue.  Thanks.

Re: [PATCH v3 5/9] i5500-git-daemon.sh: use compile-able version of Git without OpenSSL

[Junio C Hamano]

Jeff King <peff@peff.net> writes:

> How about this instead?
>
> -- >8 --
> Subject: [PATCH] t/interop: allow per-version make options
> ...
>      imap-send.c to barf, because it declares a fallback typdef for SSL.

"typedef".

> diff --git a/t/interop/i5500-git-daemon.sh b/t/interop/i5500-git-daemon.sh
> index 4d22e42f84..88712d1b5d 100755
> --- a/t/interop/i5500-git-daemon.sh
> +++ b/t/interop/i5500-git-daemon.sh
> @@ -2,6 +2,7 @@
>  
>  VERSION_A=.
>  VERSION_B=v1.0.0
> +MAKE_OPTS_B="NO_OPENSSL=TooOld"
>  
>  : ${LIB_GIT_DAEMON_PORT:=5500}
>  LIB_GIT_DAEMON_COMMAND='git.a daemon'
> diff --git a/t/interop/interop-lib.sh b/t/interop/interop-lib.sh
> index 62f4481b6e..1b5864d2a7 100644
> --- a/t/interop/interop-lib.sh
> +++ b/t/interop/interop-lib.sh
> @@ -45,7 +45,7 @@ build_version () {
>  
>  	(
>  		cd "$dir" &&
> -		make $GIT_INTEROP_MAKE_OPTS >&2 &&
> +		make $2 $GIT_INTEROP_MAKE_OPTS >&2 &&

The build options should be simple enough and this should do for now
(and when it becomes needed, it is easy to add an eval around it).

The use of $GIT_INTEROP_MAKE_OPTS here looks a bit curious.  It
overrides what the inidividual script gave in MAKE_OPTS_{A,B} and
what is globally given in GIT_INTEROP_MAKE_OPTS_{A,B}.

With this design, the following is not what we should write:

    # by default we use the frotz feature
    GIT_INTEROP_MAKE_OPTS=USE_FROTZ=YesPlease
    # but version A is too old for it
    MAKE_OPTS_A=USE_FROTZ=NoThanks
    # we do not need any cutomization for version B
    MAKE_OPTS_B=

Rather we would want to say:

    # the default should say nothing conflicting with A or B
    GIT_INTEROP_MAKE_OPTS=
    # version A is too old to use the frotz feature
    MAKE_OPTS_A=USE_FROTZ=NoThanks
    # version B is OK
    MAKE_OPTS_B=USE_FROTZ=YesPlease

As long as it is understood that GIT_INTEROP_MAKE_OPTS and *_{A,B}
are *not* meant to be used in a way for one to give default and the
other to override the defautl, but they are to give orthogonal
settings, this is fine.

>  		touch .built
>  	) || return 1
>  
> @@ -76,9 +76,11 @@ generate_wrappers () {
>  
>  VERSION_A=${GIT_TEST_VERSION_A:-$VERSION_A}
>  VERSION_B=${GIT_TEST_VERSION_B:-$VERSION_B}
> +MAKE_OPTS_A=${GIT_INTEROP_MAKE_OPTS_A:-$MAKE_OPTS_A}
> +MAKE_OPTS_B=${GIT_INTEROP_MAKE_OPTS_B:-$MAKE_OPTS_B}

Among the variables we see around here, GIT_INEROP_MAKE_OPTS
is the only one that is recorded in the GIT-BUILD-OPTIONS file,
which is included in t/interop/interop-lib.sh file.  Shouldn't
we record GIT_INEROP_MAKE_OPTS_{A,B} as well?

> -if ! DIR_A=$(build_version "$VERSION_A") ||
> -   ! DIR_B=$(build_version "$VERSION_B")
> +if ! DIR_A=$(build_version "$VERSION_A" "$MAKE_OPTS_A") ||
> +   ! DIR_B=$(build_version "$VERSION_B" "$MAKE_OPTS_B")
>  then
>  	echo >&2 "fatal: unable to build git versions"
>  	exit 1

Thanks.

Re: [PATCH v3 5/9] i5500-git-daemon.sh: use compile-able version of Git without OpenSSL

[Jeff King]

On Wed, Sep 11, 2024 at 08:28:37AM -0700, Junio C Hamano wrote:

> > -		make $GIT_INTEROP_MAKE_OPTS >&2 &&
> > +		make $2 $GIT_INTEROP_MAKE_OPTS >&2 &&
> 
> The build options should be simple enough and this should do for now
> (and when it becomes needed, it is easy to add an eval around it).
> 
> The use of $GIT_INTEROP_MAKE_OPTS here looks a bit curious.  It
> overrides what the inidividual script gave in MAKE_OPTS_{A,B} and
> what is globally given in GIT_INTEROP_MAKE_OPTS_{A,B}.
> 
> With this design, the following is not what we should write:
> 
>     # by default we use the frotz feature
>     GIT_INTEROP_MAKE_OPTS=USE_FROTZ=YesPlease
>     # but version A is too old for it
>     MAKE_OPTS_A=USE_FROTZ=NoThanks
>     # we do not need any cutomization for version B
>     MAKE_OPTS_B=
> 
> Rather we would want to say:
> 
>     # the default should say nothing conflicting with A or B
>     GIT_INTEROP_MAKE_OPTS=
>     # version A is too old to use the frotz feature
>     MAKE_OPTS_A=USE_FROTZ=NoThanks
>     # version B is OK
>     MAKE_OPTS_B=USE_FROTZ=YesPlease
> 
> As long as it is understood that GIT_INTEROP_MAKE_OPTS and *_{A,B}
> are *not* meant to be used in a way for one to give default and the
> other to override the defautl, but they are to give orthogonal
> settings, this is fine.

Yes, there are really three levels: what your platform needs for every
version, what the script asks about for its specific version, and what
you override for that specific version. So arguably the "best" order is:

  MAKE_OPTS_A < GIT_INTEROP_MAKE_OPTS < GIT_INTEROP_MAKE_OPTS_A

which always puts your preferences in front of the script's defaults,
but still lets you do a per-script override. But it didn't seem worth
the complexity to implement that. I mostly left GIT_INTEROP_MAKE_OPTS_A
as an escape hatch if you are testing an alternate version from what's
in the script, and I doubt anybody will need it at all (in all these
years I have only used it to set NO_OPENSSL for this exact case, and
judging by the lack of other people mentioning this issue I suspect
hardly anybody else has ever even run these tests).

> > @@ -76,9 +76,11 @@ generate_wrappers () {
> >  
> >  VERSION_A=${GIT_TEST_VERSION_A:-$VERSION_A}
> >  VERSION_B=${GIT_TEST_VERSION_B:-$VERSION_B}
> > +MAKE_OPTS_A=${GIT_INTEROP_MAKE_OPTS_A:-$MAKE_OPTS_A}
> > +MAKE_OPTS_B=${GIT_INTEROP_MAKE_OPTS_B:-$MAKE_OPTS_B}
> 
> Among the variables we see around here, GIT_INEROP_MAKE_OPTS
> is the only one that is recorded in the GIT-BUILD-OPTIONS file,
> which is included in t/interop/interop-lib.sh file.  Shouldn't
> we record GIT_INEROP_MAKE_OPTS_{A,B} as well?

No, I don't think that would make sense. Everything in
GIT-BUILD-OPTIONS, including GIT_INTEROP_MAKE_OPTS, is going to apply to
_all_ scripts. These _A and _B variants will vary based on individual
scripts. It's possible you might try to run the whole suite between two
specific versions, but then you'd set up GIT_INTEROP_MAKE_OPTS_{A,B} in
the environment (as you already have to do for VERSION_{A,B}).

-Peff

[PATCH v3 6/9] sha1: do not redefine `platform_SHA_CTX` and friends

[Taylor Blau]

Our in-tree SHA-1 wrappers all define platform_SHA_CTX and related
macros to point at the opaque "context" type, init, update, and similar
functions for each specific implementation.

In hash.h, we use these platform_ variables to set up the function
pointers for, e.g., the_hash_algo->init_fn(), etc.

But while these header files have a header-specific macro that prevents
them declaring their structs / functions multiple times, they
unconditionally define the platform variables, making it impossible to
load multiple SHA-1 implementations at once.

As a prerequisite for loading a separate SHA-1 implementation for
non-cryptographic uses, only define the platform_ variables if they have
not already been defined.

Signed-off-by: Taylor Blau <me@ttaylorr.com>
---
 block-sha1/sha1.h | 2 ++
 sha1/openssl.h    | 2 ++
 sha1dc_git.h      | 3 +++
 3 files changed, 7 insertions(+)

diff --git a/block-sha1/sha1.h b/block-sha1/sha1.h
index 9fb0441b988..47bb9166368 100644
--- a/block-sha1/sha1.h
+++ b/block-sha1/sha1.h
@@ -16,7 +16,9 @@ void blk_SHA1_Init(blk_SHA_CTX *ctx);
 void blk_SHA1_Update(blk_SHA_CTX *ctx, const void *dataIn, size_t len);
 void blk_SHA1_Final(unsigned char hashout[20], blk_SHA_CTX *ctx);
 
+#ifndef platform_SHA_CTX
 #define platform_SHA_CTX	blk_SHA_CTX
 #define platform_SHA1_Init	blk_SHA1_Init
 #define platform_SHA1_Update	blk_SHA1_Update
 #define platform_SHA1_Final	blk_SHA1_Final
+#endif
diff --git a/sha1/openssl.h b/sha1/openssl.h
index 006c1f4ba54..1038af47daf 100644
--- a/sha1/openssl.h
+++ b/sha1/openssl.h
@@ -40,10 +40,12 @@ static inline void openssl_SHA1_Clone(struct openssl_SHA1_CTX *dst,
 	EVP_MD_CTX_copy_ex(dst->ectx, src->ectx);
 }
 
+#ifndef platform_SHA_CTX
 #define platform_SHA_CTX openssl_SHA1_CTX
 #define platform_SHA1_Init openssl_SHA1_Init
 #define platform_SHA1_Clone openssl_SHA1_Clone
 #define platform_SHA1_Update openssl_SHA1_Update
 #define platform_SHA1_Final openssl_SHA1_Final
+#endif
 
 #endif /* SHA1_OPENSSL_H */
diff --git a/sha1dc_git.h b/sha1dc_git.h
index 60e3ce84395..f6f880cabea 100644
--- a/sha1dc_git.h
+++ b/sha1dc_git.h
@@ -18,7 +18,10 @@ void git_SHA1DCFinal(unsigned char [20], SHA1_CTX *);
 void git_SHA1DCUpdate(SHA1_CTX *ctx, const void *data, unsigned long len);
 
 #define platform_SHA_IS_SHA1DC /* used by "test-tool sha1-is-sha1dc" */
+
+#ifndef platform_SHA_CTX
 #define platform_SHA_CTX SHA1_CTX
 #define platform_SHA1_Init git_SHA1DCInit
 #define platform_SHA1_Update git_SHA1DCUpdate
 #define platform_SHA1_Final git_SHA1DCFinal
+#endif
-- 
2.46.0.430.gca674632b70

[PATCH v3 7/9] hash.h: scaffolding for _fast hashing variants

[Taylor Blau]

Git's default SHA-1 implementation is collision-detecting, which hardens
us against known SHA-1 attacks against Git objects. This makes Git
object writes safer at the expense of some speed when hashing through
the collision-detecting implementation, which is slower than
non-collision detecting alternatives.

Prepare for loading a separate "fast" SHA-1 implementation that can be
used for non-cryptographic purposes, like computing the checksum of
files that use the hashwrite() API.

This commit does not actually introduce any new compile-time knobs to
control which implementation is used as the fast SHA-1 variant, but does
add scaffolding so that the "git_hash_algo" structure has five new
function pointers which are "fast" variants of the five existing
hashing-related function pointers:

  - git_hash_init_fn fast_init_fn
  - git_hash_clone_fn fast_clone_fn
  - git_hash_update_fn fast_update_fn
  - git_hash_final_fn fast_final_fn
  - git_hash_final_oid_fn fast_final_oid_fn

The following commit will introduce compile-time knobs to specify which
SHA-1 implementation is used for non-cryptographic uses.

Signed-off-by: Taylor Blau <me@ttaylorr.com>
---
 hash.h        | 42 ++++++++++++++++++++++++++++++++++++++++++
 object-file.c | 42 ++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 84 insertions(+)

diff --git a/hash.h b/hash.h
index 72ffbc862e5..5e5b8205b58 100644
--- a/hash.h
+++ b/hash.h
@@ -44,14 +44,32 @@
 #define platform_SHA1_Final    	SHA1_Final
 #endif
 
+#ifndef platform_SHA_CTX_fast
+#  define platform_SHA_CTX_fast     platform_SHA_CTX
+#  define platform_SHA1_Init_fast   platform_SHA1_Init
+#  define platform_SHA1_Update_fast platform_SHA1_Update
+#  define platform_SHA1_Final_fast  platform_SHA1_Final
+#  ifdef platform_SHA1_Clone
+#    define platform_SHA1_Clone_fast platform_SHA1_Clone
+#  endif
+#endif
+
 #define git_SHA_CTX		platform_SHA_CTX
 #define git_SHA1_Init		platform_SHA1_Init
 #define git_SHA1_Update		platform_SHA1_Update
 #define git_SHA1_Final		platform_SHA1_Final
 
+#define git_SHA_CTX_fast	platform_SHA_CTX_fast
+#define git_SHA1_Init_fast	platform_SHA1_Init_fast
+#define git_SHA1_Update_fast	platform_SHA1_Update_fast
+#define git_SHA1_Final_fast	platform_SHA1_Final_fast
+
 #ifdef platform_SHA1_Clone
 #define git_SHA1_Clone	platform_SHA1_Clone
 #endif
+#ifdef platform_SHA1_Clone_fast
+#  define git_SHA1_Clone_fast platform_SHA1_Clone_fast
+#endif
 
 #ifndef platform_SHA256_CTX
 #define platform_SHA256_CTX	SHA256_CTX
@@ -81,6 +99,13 @@ static inline void git_SHA1_Clone(git_SHA_CTX *dst, const git_SHA_CTX *src)
 	memcpy(dst, src, sizeof(*dst));
 }
 #endif
+#ifndef SHA1_NEEDS_CLONE_HELPER_FAST
+static inline void git_SHA1_Clone_fast(git_SHA_CTX_fast *dst,
+				       const git_SHA_CTX_fast *src)
+{
+	memcpy(dst, src, sizeof(*dst));
+}
+#endif
 
 #ifndef SHA256_NEEDS_CLONE_HELPER
 static inline void git_SHA256_Clone(git_SHA256_CTX *dst, const git_SHA256_CTX *src)
@@ -178,6 +203,8 @@ enum get_oid_result {
 /* A suitably aligned type for stack allocations of hash contexts. */
 union git_hash_ctx {
 	git_SHA_CTX sha1;
+	git_SHA_CTX_fast sha1_fast;
+
 	git_SHA256_CTX sha256;
 };
 typedef union git_hash_ctx git_hash_ctx;
@@ -222,6 +249,21 @@ struct git_hash_algo {
 	/* The hash finalization function for object IDs. */
 	git_hash_final_oid_fn final_oid_fn;
 
+	/* The fast / non-cryptographic hash initialization function. */
+	git_hash_init_fn fast_init_fn;
+
+	/* The fast / non-cryptographic hash context cloning function. */
+	git_hash_clone_fn fast_clone_fn;
+
+	/* The fast / non-cryptographic hash update function. */
+	git_hash_update_fn fast_update_fn;
+
+	/* The fast / non-cryptographic hash finalization function. */
+	git_hash_final_fn fast_final_fn;
+
+	/* The fast / non-cryptographic hash finalization function. */
+	git_hash_final_oid_fn fast_final_oid_fn;
+
 	/* The OID of the empty tree. */
 	const struct object_id *empty_tree;
 
diff --git a/object-file.c b/object-file.c
index 85f91516429..84dd7d0fab6 100644
--- a/object-file.c
+++ b/object-file.c
@@ -115,6 +115,33 @@ static void git_hash_sha1_final_oid(struct object_id *oid, git_hash_ctx *ctx)
 	oid->algo = GIT_HASH_SHA1;
 }
 
+static void git_hash_sha1_init_fast(git_hash_ctx *ctx)
+{
+	git_SHA1_Init_fast(&ctx->sha1_fast);
+}
+
+static void git_hash_sha1_clone_fast(git_hash_ctx *dst, const git_hash_ctx *src)
+{
+	git_SHA1_Clone_fast(&dst->sha1_fast, &src->sha1_fast);
+}
+
+static void git_hash_sha1_update_fast(git_hash_ctx *ctx, const void *data,
+				      size_t len)
+{
+	git_SHA1_Update_fast(&ctx->sha1_fast, data, len);
+}
+
+static void git_hash_sha1_final_fast(unsigned char *hash, git_hash_ctx *ctx)
+{
+	git_SHA1_Final_fast(hash, &ctx->sha1_fast);
+}
+
+static void git_hash_sha1_final_oid_fast(struct object_id *oid, git_hash_ctx *ctx)
+{
+	git_SHA1_Final_fast(oid->hash, &ctx->sha1_fast);
+	memset(oid->hash + GIT_SHA1_RAWSZ, 0, GIT_MAX_RAWSZ - GIT_SHA1_RAWSZ);
+	oid->algo = GIT_HASH_SHA1;
+}
 
 static void git_hash_sha256_init(git_hash_ctx *ctx)
 {
@@ -189,6 +216,11 @@ const struct git_hash_algo hash_algos[GIT_HASH_NALGOS] = {
 		.update_fn = git_hash_unknown_update,
 		.final_fn = git_hash_unknown_final,
 		.final_oid_fn = git_hash_unknown_final_oid,
+		.fast_init_fn = git_hash_unknown_init,
+		.fast_clone_fn = git_hash_unknown_clone,
+		.fast_update_fn = git_hash_unknown_update,
+		.fast_final_fn = git_hash_unknown_final,
+		.fast_final_oid_fn = git_hash_unknown_final_oid,
 		.empty_tree = NULL,
 		.empty_blob = NULL,
 		.null_oid = NULL,
@@ -204,6 +236,11 @@ const struct git_hash_algo hash_algos[GIT_HASH_NALGOS] = {
 		.update_fn = git_hash_sha1_update,
 		.final_fn = git_hash_sha1_final,
 		.final_oid_fn = git_hash_sha1_final_oid,
+		.fast_init_fn = git_hash_sha1_init_fast,
+		.fast_clone_fn = git_hash_sha1_clone_fast,
+		.fast_update_fn = git_hash_sha1_update_fast,
+		.fast_final_fn = git_hash_sha1_final_fast,
+		.fast_final_oid_fn = git_hash_sha1_final_oid_fast,
 		.empty_tree = &empty_tree_oid,
 		.empty_blob = &empty_blob_oid,
 		.null_oid = &null_oid_sha1,
@@ -219,6 +256,11 @@ const struct git_hash_algo hash_algos[GIT_HASH_NALGOS] = {
 		.update_fn = git_hash_sha256_update,
 		.final_fn = git_hash_sha256_final,
 		.final_oid_fn = git_hash_sha256_final_oid,
+		.fast_init_fn = git_hash_sha256_init,
+		.fast_clone_fn = git_hash_sha256_clone,
+		.fast_update_fn = git_hash_sha256_update,
+		.fast_final_fn = git_hash_sha256_final,
+		.fast_final_oid_fn = git_hash_sha256_final_oid,
 		.empty_tree = &empty_tree_oid_sha256,
 		.empty_blob = &empty_blob_oid_sha256,
 		.null_oid = &null_oid_sha256,
-- 
2.46.0.430.gca674632b70

[PATCH v3 8/9] Makefile: allow specifying a SHA-1 for non-cryptographic uses

[Taylor Blau]

Introduce _FAST variants of the OPENSSL_SHA1, BLK_SHA1, and
APPLE_COMMON_CRYPTO_SHA1 compile-time knobs which indicate which SHA-1
implementation is to be used for non-cryptographic uses.

There are a couple of small implementation notes worth mentioning:

  - There is no way to select the collision detecting SHA-1 as the
    "fast" fallback, since the fast fallback is only for
    non-cryptographic uses, and is meant to be faster than our
    collision-detecting implementation.

  - There are no similar knobs for SHA-256, since no collision attacks
    are presently known and thus no collision-detecting implementations
    actually exist.

Signed-off-by: Taylor Blau <me@ttaylorr.com>
---
 Makefile | 25 +++++++++++++++++++++++++
 hash.h   | 30 ++++++++++++++++++++++++++++++
 2 files changed, 55 insertions(+)

diff --git a/Makefile b/Makefile
index e298c8b55ec..d24f9088802 100644
--- a/Makefile
+++ b/Makefile
@@ -517,6 +517,10 @@ include shared.mak
 # Define APPLE_COMMON_CRYPTO_SHA1 to use Apple's CommonCrypto for
 # SHA-1.
 #
+# Define the same Makefile knobs as above, but suffixed with _FAST to
+# use the corresponding implementations for "fast" SHA-1 hashing for
+# non-cryptographic purposes.
+#
 # If don't enable any of the *_SHA1 settings in this section, Git will
 # default to its built-in sha1collisiondetection library, which is a
 # collision-detecting sha1 This is slower, but may detect attempted
@@ -1982,6 +1986,27 @@ endif
 endif
 endif
 
+ifdef OPENSSL_SHA1_FAST
+ifndef OPENSSL_SHA1
+	EXTLIBS += $(LIB_4_CRYPTO)
+	BASIC_CFLAGS += -DSHA1_OPENSSL_FAST
+endif
+else
+ifdef BLK_SHA1_FAST
+ifndef BLK_SHA1
+	LIB_OBJS += block-sha1/sha1.o
+	BASIC_CFLAGS += -DSHA1_BLK_FAST
+endif
+else
+ifdef APPLE_COMMON_CRYPTO_SHA1_FAST
+ifndef APPLE_COMMON_CRYPTO_SHA1
+	COMPAT_CFLAGS += -DCOMMON_DIGEST_FOR_OPENSSL
+	BASIC_CFLAGS += -DSHA1_APPLE_FAST
+endif
+endif
+endif
+endif
+
 ifdef OPENSSL_SHA256
 	EXTLIBS += $(LIB_4_CRYPTO)
 	BASIC_CFLAGS += -DSHA256_OPENSSL
diff --git a/hash.h b/hash.h
index 5e5b8205b58..6ea0d968017 100644
--- a/hash.h
+++ b/hash.h
@@ -15,6 +15,36 @@
 #include "block-sha1/sha1.h"
 #endif
 
+#if defined(SHA1_APPLE_FAST)
+#  include <CommonCrypto/CommonDigest.h>
+#  define platform_SHA_CTX_fast CC_SHA1_CTX
+#  define platform_SHA1_Init_fast CC_SHA1_Init
+#  define platform_SHA1_Update_fast CC_SHA1_Update
+#  define platform_SHA1_Final_fast CC_SHA1_Final
+#elif defined(SHA1_OPENSSL_FAST)
+#  include <openssl/sha.h>
+#  if defined(OPENSSL_API_LEVEL) && OPENSSL_API_LEVEL >= 3
+#    define SHA1_NEEDS_CLONE_HELPER_FAST
+#    include "sha1/openssl.h"
+#    define platform_SHA_CTX_fast openssl_SHA1_CTX
+#    define platform_SHA1_Init_fast openssl_SHA1_Init
+#    define platform_SHA1_Clone_fast openssl_SHA1_Clone
+#    define platform_SHA1_Update_fast openssl_SHA1_Update
+#    define platform_SHA1_Final_fast openssl_SHA1_Final
+#  else
+#    define platform_SHA_CTX_fast SHA_CTX
+#    define platform_SHA1_Init_fast SHA1_Init
+#    define platform_SHA1_Update_fast SHA1_Update
+#    define platform_SHA1_Final_fast SHA1_Final
+#  endif
+#elif defined(SHA1_BLK_FAST)
+#  include "block-sha1/sha1.h"
+#  define platform_SHA_CTX_fast blk_SHA_CTX
+#  define platform_SHA1_Init_fast blk_SHA1_Init
+#  define platform_SHA1_Update_fast blk_SHA1_Update
+#  define platform_SHA1_Final_fast blk_SHA1_Final
+#endif
+
 #if defined(SHA256_NETTLE)
 #include "sha256/nettle.h"
 #elif defined(SHA256_GCRYPT)
-- 
2.46.0.430.gca674632b70

[PATCH v3 9/9] csum-file.c: use fast SHA-1 implementation when available

[Taylor Blau]

Update hashwrite() and friends to use the fast_-variants of hashing
functions, calling for e.g., "the_hash_algo->fast_update_fn()" instead
of "the_hash_algo->update_fn()".

These callers only use the_hash_algo to produce a checksum, which we
depend on for data integrity, but not for cryptographic purposes, so
these callers are safe to use the fast (and potentially non-collision
detecting) SHA-1 implementation.

To time this, I took a freshly packed copy of linux.git, and ran the
following with and without the OPENSSL_SHA1_FAST=1 build-knob. Both
versions were compiled with -O3:

    $ git for-each-ref --format='%(objectname)' refs/heads refs/tags >in
    $ valgrind --tool=callgrind ~/src/git/git-pack-objects \
        --revs --stdout --all-progress --use-bitmap-index <in >/dev/null

Without OPENSSL_SHA1_FAST=1 (that is, using the collision-detecting
SHA-1 implementation for both cryptographic and non-cryptographic
purposes), we spend a significant amount of our instruction count in
hashwrite():

    $ callgrind_annotate --inclusive=yes | grep hashwrite | head -n1
    159,998,868,413 (79.42%)  /home/ttaylorr/src/git/csum-file.c:hashwrite [/home/ttaylorr/src/git/git-pack-objects]

, and the resulting "clone" takes 19.219 seconds of wall clock time,
18.94 seconds of user time and 0.28 seconds of system time.

Compiling with OPENSSL_SHA1_FAST=1, we spend ~60% fewer instructions in
hashwrite():

    $ callgrind_annotate --inclusive=yes | grep hashwrite | head -n1
     59,164,001,176 (58.79%)  /home/ttaylorr/src/git/csum-file.c:hashwrite [/home/ttaylorr/src/git/git-pack-objects]

, and generate the resulting "clone" much faster, in only 11.597 seconds
of wall time, 11.37 seconds of user time, and 0.23 seconds of system
time, for a ~40% speed-up.

Signed-off-by: Taylor Blau <me@ttaylorr.com>
---
 csum-file.c | 18 +++++++++---------
 1 file changed, 9 insertions(+), 9 deletions(-)

diff --git a/csum-file.c b/csum-file.c
index bf82ad8f9f5..cb8c39ecf3a 100644
--- a/csum-file.c
+++ b/csum-file.c
@@ -50,7 +50,7 @@ void hashflush(struct hashfile *f)
 
 	if (offset) {
 		if (!f->skip_hash)
-			the_hash_algo->update_fn(&f->ctx, f->buffer, offset);
+			the_hash_algo->fast_update_fn(&f->ctx, f->buffer, offset);
 		flush(f, f->buffer, offset);
 		f->offset = 0;
 	}
@@ -73,7 +73,7 @@ int finalize_hashfile(struct hashfile *f, unsigned char *result,
 	if (f->skip_hash)
 		hashclr(f->buffer, the_repository->hash_algo);
 	else
-		the_hash_algo->final_fn(f->buffer, &f->ctx);
+		the_hash_algo->fast_final_fn(f->buffer, &f->ctx);
 
 	if (result)
 		hashcpy(result, f->buffer, the_repository->hash_algo);
@@ -128,7 +128,7 @@ void hashwrite(struct hashfile *f, const void *buf, unsigned int count)
 			 * f->offset is necessarily zero.
 			 */
 			if (!f->skip_hash)
-				the_hash_algo->update_fn(&f->ctx, buf, nr);
+				the_hash_algo->fast_update_fn(&f->ctx, buf, nr);
 			flush(f, buf, nr);
 		} else {
 			/*
@@ -174,7 +174,7 @@ static struct hashfile *hashfd_internal(int fd, const char *name,
 	f->name = name;
 	f->do_crc = 0;
 	f->skip_hash = 0;
-	the_hash_algo->init_fn(&f->ctx);
+	the_hash_algo->fast_init_fn(&f->ctx);
 
 	f->buffer_len = buffer_len;
 	f->buffer = xmalloc(buffer_len);
@@ -208,7 +208,7 @@ void hashfile_checkpoint(struct hashfile *f, struct hashfile_checkpoint *checkpo
 {
 	hashflush(f);
 	checkpoint->offset = f->total;
-	the_hash_algo->clone_fn(&checkpoint->ctx, &f->ctx);
+	the_hash_algo->fast_clone_fn(&checkpoint->ctx, &f->ctx);
 }
 
 int hashfile_truncate(struct hashfile *f, struct hashfile_checkpoint *checkpoint)
@@ -219,7 +219,7 @@ int hashfile_truncate(struct hashfile *f, struct hashfile_checkpoint *checkpoint
 	    lseek(f->fd, offset, SEEK_SET) != offset)
 		return -1;
 	f->total = offset;
-	the_hash_algo->clone_fn(&f->ctx, &checkpoint->ctx);
+	the_hash_algo->fast_clone_fn(&f->ctx, &checkpoint->ctx);
 	f->offset = 0; /* hashflush() was called in checkpoint */
 	return 0;
 }
@@ -245,9 +245,9 @@ int hashfile_checksum_valid(const unsigned char *data, size_t total_len)
 	if (total_len < the_hash_algo->rawsz)
 		return 0; /* say "too short"? */
 
-	the_hash_algo->init_fn(&ctx);
-	the_hash_algo->update_fn(&ctx, data, data_len);
-	the_hash_algo->final_fn(got, &ctx);
+	the_hash_algo->fast_init_fn(&ctx);
+	the_hash_algo->fast_update_fn(&ctx, data, data_len);
+	the_hash_algo->fast_final_fn(got, &ctx);
 
 	return hasheq(got, data + data_len, the_repository->hash_algo);
 }
-- 
2.46.0.430.gca674632b70

Re: [PATCH v3 0/9] hash.h: support choosing a separate SHA-1 for non-cryptographic uses

[Junio C Hamano]

Taylor Blau <me@ttaylorr.com> writes:

> Peff and I wrote the first four (new) patches together, and I feel
> confident now that we're in a good spot to address the concerns raised
> in [1].

I just read the first four and they made sense.

With "fast" -> "insecure", the latter half of the series would also
cover the review comments on earlier rounds, I would imagine?

Thanks.

[PATCH v4 0/8] hash.h: support choosing a separate SHA-1 for non-cryptographic uses

[Taylor Blau]

Here is a reroll of mine and Peff's series to add a build-time knob to
allow selecting an alternative SHA-1 implementation for
non-cryptographic hashing within Git, starting with the `hashwrite()`
family of functions.

This version is another fairly substantial reroll, with the main
differences being renaming the "fast" SHA-1 options to "unsafe", as well
as not running the collision checks via finalize_object_file() when
handling loose objects (see the relevant patches for details on why).

Note also there is an important bug fix in finalize_object_file() to
unlink() the temporary file when we do run the collision check, but no
collisions were found. This bug was causing a pile-up of tmp_obj_XXXXXX
files in GitHub's infrastructure.

Thanks in advance for your review!

Taylor Blau (8):
  finalize_object_file(): check for name collision before renaming
  finalize_object_file(): refactor unlink_or_warn() placement
  finalize_object_file(): implement collision check
  pack-objects: use finalize_object_file() to rename pack/idx/etc
  sha1: do not redefine `platform_SHA_CTX` and friends
  hash.h: scaffolding for _unsafe hashing variants
  Makefile: allow specifying a SHA-1 for non-cryptographic uses
  csum-file.c: use unsafe SHA-1 implementation when available

 Makefile                              |  25 ++++++
 block-sha1/sha1.h                     |   2 +
 csum-file.c                           |  18 ++--
 hash.h                                |  72 +++++++++++++++
 object-file.c                         | 124 ++++++++++++++++++++++++--
 object-file.h                         |   6 ++
 pack-write.c                          |   7 +-
 sha1/openssl.h                        |   2 +
 sha1dc_git.h                          |   3 +
 t/t5303-pack-corruption-resilience.sh |   7 +-
 tmp-objdir.c                          |  26 ++++--
 11 files changed, 266 insertions(+), 26 deletions(-)

Range-diff against v3:
 1:  738b1eb17b4 =  1:  6f1ee91fff3 finalize_object_file(): check for name collision before renaming
 2:  e1c2c39711f =  2:  133047ca8c9 finalize_object_file(): refactor unlink_or_warn() placement
 3:  0feee5d1d4f !  3:  ed9eeef8513 finalize_object_file(): implement collision check
    @@ Commit message
     
         The new check will cause the write of new differing content to be a
         failure, rather than a silent noop, and we'll retain the temporary file
    -    on disk.
    +    on disk. If there's no collision present, we'll clean up the temporary
    +    file as usual after either rename()-ing or link()-ing it into place.
     
         Note that this may cause some extra computation when the files are in
    -    fact identical, but this should happen rarely. For example, when writing
    -    a loose object, we compute the object id first, then check manually for
    -    an existing object (so that we can freshen its timestamp) before
    -    actually trying to write and link the new data.
    +    fact identical, but this should happen rarely.
    +
    +    Loose objects are exempt from this check, and the collision check may be
    +    skipped by calling the _flags variant of this function with the
    +    FOF_SKIP_COLLISION_CHECK bit set. This is done for a couple of reasons:
    +
    +      - We don't treat the hash of the loose object file's contents as a
    +        checksum, since the same loose object can be stored using different
    +        bytes on disk (e.g., when adjusting core.compression, using a
    +        different version of zlib, etc.).
    +
    +        This is fundamentally different from cases where
    +        finalize_object_file() is operating over a file which uses the hash
    +        value as a checksum of the contents. In other words, a pair of
    +        identical loose objects can be stored using different bytes on disk,
    +        and that should not be treated as a collision.
    +
    +      - We already use the path of the loose object as its hash value /
    +        object name, so checking for collisions at the content level doesn't
    +        add anything.
    +
    +        This is why we do not bother to check the inflated object contents
    +        for collisions either, since either (a) the object contents have the
    +        fingerprint of a SHA-1 collision, in which case the collision
    +        detecting SHA-1 implementation used to hash the contents to give us
    +        a path would have already rejected it, or (b) the contents are part
    +        of a colliding pair which does not bear the same fingerprints of
    +        known collision attacks, in which case we would not have caught it
    +        anyway.
    +
    +        So skipping the collision check here does not change for better or
    +        worse the hardness of loose object writes.
    +
    +    As a small note related to the latter bullet point above, we must teach
    +    the tmp-objdir routines to similarly skip the content-level collision
    +    checks when calling migrate_one() on a loose object file, which we do by
    +    setting the FOF_SKIP_COLLISION_CHECK bit when we are inside of a loose
    +    object shard.
     
         Co-authored-by: Jeff King <peff@peff.net>
         Signed-off-by: Jeff King <peff@peff.net>
    @@ object-file.c: static void write_object_file_prepare_literally(const struct git_
      /*
       * Move the just written object into its final resting place.
       */
    + int finalize_object_file(const char *tmpfile, const char *filename)
    ++{
    ++	return finalize_object_file_flags(tmpfile, filename, 0);
    ++}
    ++
    ++int finalize_object_file_flags(const char *tmpfile, const char *filename,
    ++			       enum finalize_object_file_flags flags)
    + {
    + 	struct stat st;
    + 	int ret = 0;
     @@ object-file.c: int finalize_object_file(const char *tmpfile, const char *filename)
      			errno = saved_errno;
      			return error_errno(_("unable to write file %s"), filename);
      		}
     -		/* FIXME!!! Collision check here ? */
    --		unlink_or_warn(tmpfile);
    -+		if (check_collision(tmpfile, filename))
    -+			return -1;
    ++		if (!(flags & FOF_SKIP_COLLISION_CHECK) &&
    ++		    check_collision(tmpfile, filename))
    ++				return -1;
    + 		unlink_or_warn(tmpfile);
      	}
      
    - out:
    +@@ object-file.c: static int write_loose_object(const struct object_id *oid, char *hdr,
    + 			warning_errno(_("failed utime() on %s"), tmp_file.buf);
    + 	}
    + 
    +-	return finalize_object_file(tmp_file.buf, filename.buf);
    ++	return finalize_object_file_flags(tmp_file.buf, filename.buf,
    ++					  FOF_SKIP_COLLISION_CHECK);
    + }
    + 
    + static int freshen_loose_object(const struct object_id *oid)
    +@@ object-file.c: int stream_loose_object(struct input_stream *in_stream, size_t len,
    + 		strbuf_release(&dir);
    + 	}
    + 
    +-	err = finalize_object_file(tmp_file.buf, filename.buf);
    ++	err = finalize_object_file_flags(tmp_file.buf, filename.buf,
    ++					 FOF_SKIP_COLLISION_CHECK);
    + 	if (!err && compat)
    + 		err = repo_add_loose_object_map(the_repository, oid, &compat_oid);
    + cleanup:
    +
    + ## object-file.h ##
    +@@ object-file.h: int check_object_signature(struct repository *r, const struct object_id *oid,
    +  */
    + int stream_object_signature(struct repository *r, const struct object_id *oid);
    + 
    ++enum finalize_object_file_flags {
    ++	FOF_SKIP_COLLISION_CHECK = 1,
    ++};
    ++
    + int finalize_object_file(const char *tmpfile, const char *filename);
    ++int finalize_object_file_flags(const char *tmpfile, const char *filename,
    ++			       enum finalize_object_file_flags flags);
    + 
    + /* Helper to check and "touch" a file */
    + int check_and_freshen_file(const char *fn, int freshen);
    +
    + ## tmp-objdir.c ##
    +@@ tmp-objdir.c: static int read_dir_paths(struct string_list *out, const char *path)
    + 	return 0;
    + }
    + 
    +-static int migrate_paths(struct strbuf *src, struct strbuf *dst);
    ++static int migrate_paths(struct strbuf *src, struct strbuf *dst,
    ++			 enum finalize_object_file_flags flags);
    + 
    +-static int migrate_one(struct strbuf *src, struct strbuf *dst)
    ++static int migrate_one(struct strbuf *src, struct strbuf *dst,
    ++		       enum finalize_object_file_flags flags)
    + {
    + 	struct stat st;
    + 
    +@@ tmp-objdir.c: static int migrate_one(struct strbuf *src, struct strbuf *dst)
    + 				return -1;
    + 		} else if (errno != EEXIST)
    + 			return -1;
    +-		return migrate_paths(src, dst);
    ++		return migrate_paths(src, dst, flags);
    + 	}
    +-	return finalize_object_file(src->buf, dst->buf);
    ++	return finalize_object_file_flags(src->buf, dst->buf, flags);
    + }
    + 
    +-static int migrate_paths(struct strbuf *src, struct strbuf *dst)
    ++static int is_loose_object_shard(const char *name)
    ++{
    ++	return strlen(name) == 2 && isxdigit(name[0]) && isxdigit(name[1]);
    ++}
    ++
    ++static int migrate_paths(struct strbuf *src, struct strbuf *dst,
    ++			 enum finalize_object_file_flags flags)
    + {
    + 	size_t src_len = src->len, dst_len = dst->len;
    + 	struct string_list paths = STRING_LIST_INIT_DUP;
    +@@ tmp-objdir.c: static int migrate_paths(struct strbuf *src, struct strbuf *dst)
    + 
    + 	for (i = 0; i < paths.nr; i++) {
    + 		const char *name = paths.items[i].string;
    ++		enum finalize_object_file_flags flags_copy = flags;
    + 
    + 		strbuf_addf(src, "/%s", name);
    + 		strbuf_addf(dst, "/%s", name);
    + 
    +-		ret |= migrate_one(src, dst);
    ++		if (is_loose_object_shard(name))
    ++			flags_copy |= FOF_SKIP_COLLISION_CHECK;
    ++
    ++		ret |= migrate_one(src, dst, flags_copy);
    + 
    + 		strbuf_setlen(src, src_len);
    + 		strbuf_setlen(dst, dst_len);
    +@@ tmp-objdir.c: int tmp_objdir_migrate(struct tmp_objdir *t)
    + 	strbuf_addbuf(&src, &t->path);
    + 	strbuf_addstr(&dst, repo_get_object_directory(the_repository));
    + 
    +-	ret = migrate_paths(&src, &dst);
    ++	ret = migrate_paths(&src, &dst, 0);
    + 
    + 	strbuf_release(&src);
    + 	strbuf_release(&dst);
 4:  620dde48a9d =  4:  3cc7f7b1f67 pack-objects: use finalize_object_file() to rename pack/idx/etc
 5:  bfe992765cd <  -:  ----------- i5500-git-daemon.sh: use compile-able version of Git without OpenSSL
 6:  22863d9f6df =  5:  8f8ac0f5b0e sha1: do not redefine `platform_SHA_CTX` and friends
 7:  119c318d812 !  6:  d300e9c6887 hash.h: scaffolding for _fast hashing variants
    @@ Metadata
     Author: Taylor Blau <me@ttaylorr.com>
     
      ## Commit message ##
    -    hash.h: scaffolding for _fast hashing variants
    +    hash.h: scaffolding for _unsafe hashing variants
     
         Git's default SHA-1 implementation is collision-detecting, which hardens
         us against known SHA-1 attacks against Git objects. This makes Git
    @@ Commit message
         the collision-detecting implementation, which is slower than
         non-collision detecting alternatives.
     
    -    Prepare for loading a separate "fast" SHA-1 implementation that can be
    +    Prepare for loading a separate "unsafe" SHA-1 implementation that can be
         used for non-cryptographic purposes, like computing the checksum of
         files that use the hashwrite() API.
     
         This commit does not actually introduce any new compile-time knobs to
    -    control which implementation is used as the fast SHA-1 variant, but does
    -    add scaffolding so that the "git_hash_algo" structure has five new
    -    function pointers which are "fast" variants of the five existing
    +    control which implementation is used as the unsafe SHA-1 variant, but
    +    does add scaffolding so that the "git_hash_algo" structure has five new
    +    function pointers which are "unsafe" variants of the five existing
         hashing-related function pointers:
     
    -      - git_hash_init_fn fast_init_fn
    -      - git_hash_clone_fn fast_clone_fn
    -      - git_hash_update_fn fast_update_fn
    -      - git_hash_final_fn fast_final_fn
    -      - git_hash_final_oid_fn fast_final_oid_fn
    +      - git_hash_init_fn unsafe_init_fn
    +      - git_hash_clone_fn unsafe_clone_fn
    +      - git_hash_update_fn unsafe_update_fn
    +      - git_hash_final_fn unsafe_final_fn
    +      - git_hash_final_oid_fn unsafe_final_oid_fn
     
         The following commit will introduce compile-time knobs to specify which
         SHA-1 implementation is used for non-cryptographic uses.
    @@ hash.h
      #define platform_SHA1_Final    	SHA1_Final
      #endif
      
    -+#ifndef platform_SHA_CTX_fast
    -+#  define platform_SHA_CTX_fast     platform_SHA_CTX
    -+#  define platform_SHA1_Init_fast   platform_SHA1_Init
    -+#  define platform_SHA1_Update_fast platform_SHA1_Update
    -+#  define platform_SHA1_Final_fast  platform_SHA1_Final
    ++#ifndef platform_SHA_CTX_unsafe
    ++#  define platform_SHA_CTX_unsafe      platform_SHA_CTX
    ++#  define platform_SHA1_Init_unsafe    platform_SHA1_Init
    ++#  define platform_SHA1_Update_unsafe  platform_SHA1_Update
    ++#  define platform_SHA1_Final_unsafe   platform_SHA1_Final
     +#  ifdef platform_SHA1_Clone
    -+#    define platform_SHA1_Clone_fast platform_SHA1_Clone
    ++#    define platform_SHA1_Clone_unsafe platform_SHA1_Clone
     +#  endif
     +#endif
     +
    @@ hash.h
      #define git_SHA1_Update		platform_SHA1_Update
      #define git_SHA1_Final		platform_SHA1_Final
      
    -+#define git_SHA_CTX_fast	platform_SHA_CTX_fast
    -+#define git_SHA1_Init_fast	platform_SHA1_Init_fast
    -+#define git_SHA1_Update_fast	platform_SHA1_Update_fast
    -+#define git_SHA1_Final_fast	platform_SHA1_Final_fast
    ++#define git_SHA_CTX_unsafe	platform_SHA_CTX_unsafe
    ++#define git_SHA1_Init_unsafe	platform_SHA1_Init_unsafe
    ++#define git_SHA1_Update_unsafe	platform_SHA1_Update_unsafe
    ++#define git_SHA1_Final_unsafe	platform_SHA1_Final_unsafe
     +
      #ifdef platform_SHA1_Clone
      #define git_SHA1_Clone	platform_SHA1_Clone
      #endif
    -+#ifdef platform_SHA1_Clone_fast
    -+#  define git_SHA1_Clone_fast platform_SHA1_Clone_fast
    ++#ifdef platform_SHA1_Clone_unsafe
    ++#  define git_SHA1_Clone_unsafe platform_SHA1_Clone_unsafe
     +#endif
      
      #ifndef platform_SHA256_CTX
    @@ hash.h: static inline void git_SHA1_Clone(git_SHA_CTX *dst, const git_SHA_CTX *s
      	memcpy(dst, src, sizeof(*dst));
      }
      #endif
    -+#ifndef SHA1_NEEDS_CLONE_HELPER_FAST
    -+static inline void git_SHA1_Clone_fast(git_SHA_CTX_fast *dst,
    -+				       const git_SHA_CTX_fast *src)
    ++#ifndef SHA1_NEEDS_CLONE_HELPER_UNSAFE
    ++static inline void git_SHA1_Clone_unsafe(git_SHA_CTX_unsafe *dst,
    ++				       const git_SHA_CTX_unsafe *src)
     +{
     +	memcpy(dst, src, sizeof(*dst));
     +}
    @@ hash.h: enum get_oid_result {
      /* A suitably aligned type for stack allocations of hash contexts. */
      union git_hash_ctx {
      	git_SHA_CTX sha1;
    -+	git_SHA_CTX_fast sha1_fast;
    ++	git_SHA_CTX_unsafe sha1_unsafe;
     +
      	git_SHA256_CTX sha256;
      };
    @@ hash.h: struct git_hash_algo {
      	/* The hash finalization function for object IDs. */
      	git_hash_final_oid_fn final_oid_fn;
      
    -+	/* The fast / non-cryptographic hash initialization function. */
    -+	git_hash_init_fn fast_init_fn;
    ++	/* The non-cryptographic hash initialization function. */
    ++	git_hash_init_fn unsafe_init_fn;
     +
    -+	/* The fast / non-cryptographic hash context cloning function. */
    -+	git_hash_clone_fn fast_clone_fn;
    ++	/* The non-cryptographic hash context cloning function. */
    ++	git_hash_clone_fn unsafe_clone_fn;
     +
    -+	/* The fast / non-cryptographic hash update function. */
    -+	git_hash_update_fn fast_update_fn;
    ++	/* The non-cryptographic hash update function. */
    ++	git_hash_update_fn unsafe_update_fn;
     +
    -+	/* The fast / non-cryptographic hash finalization function. */
    -+	git_hash_final_fn fast_final_fn;
    ++	/* The non-cryptographic hash finalization function. */
    ++	git_hash_final_fn unsafe_final_fn;
     +
    -+	/* The fast / non-cryptographic hash finalization function. */
    -+	git_hash_final_oid_fn fast_final_oid_fn;
    ++	/* The non-cryptographic hash finalization function. */
    ++	git_hash_final_oid_fn unsafe_final_oid_fn;
     +
      	/* The OID of the empty tree. */
      	const struct object_id *empty_tree;
    @@ object-file.c: static void git_hash_sha1_final_oid(struct object_id *oid, git_ha
      	oid->algo = GIT_HASH_SHA1;
      }
      
    -+static void git_hash_sha1_init_fast(git_hash_ctx *ctx)
    ++static void git_hash_sha1_init_unsafe(git_hash_ctx *ctx)
     +{
    -+	git_SHA1_Init_fast(&ctx->sha1_fast);
    ++	git_SHA1_Init_unsafe(&ctx->sha1_unsafe);
     +}
     +
    -+static void git_hash_sha1_clone_fast(git_hash_ctx *dst, const git_hash_ctx *src)
    ++static void git_hash_sha1_clone_unsafe(git_hash_ctx *dst, const git_hash_ctx *src)
     +{
    -+	git_SHA1_Clone_fast(&dst->sha1_fast, &src->sha1_fast);
    ++	git_SHA1_Clone_unsafe(&dst->sha1_unsafe, &src->sha1_unsafe);
     +}
     +
    -+static void git_hash_sha1_update_fast(git_hash_ctx *ctx, const void *data,
    ++static void git_hash_sha1_update_unsafe(git_hash_ctx *ctx, const void *data,
     +				      size_t len)
     +{
    -+	git_SHA1_Update_fast(&ctx->sha1_fast, data, len);
    ++	git_SHA1_Update_unsafe(&ctx->sha1_unsafe, data, len);
     +}
     +
    -+static void git_hash_sha1_final_fast(unsigned char *hash, git_hash_ctx *ctx)
    ++static void git_hash_sha1_final_unsafe(unsigned char *hash, git_hash_ctx *ctx)
     +{
    -+	git_SHA1_Final_fast(hash, &ctx->sha1_fast);
    ++	git_SHA1_Final_unsafe(hash, &ctx->sha1_unsafe);
     +}
     +
    -+static void git_hash_sha1_final_oid_fast(struct object_id *oid, git_hash_ctx *ctx)
    ++static void git_hash_sha1_final_oid_unsafe(struct object_id *oid, git_hash_ctx *ctx)
     +{
    -+	git_SHA1_Final_fast(oid->hash, &ctx->sha1_fast);
    ++	git_SHA1_Final_unsafe(oid->hash, &ctx->sha1_unsafe);
     +	memset(oid->hash + GIT_SHA1_RAWSZ, 0, GIT_MAX_RAWSZ - GIT_SHA1_RAWSZ);
     +	oid->algo = GIT_HASH_SHA1;
     +}
    @@ object-file.c: const struct git_hash_algo hash_algos[GIT_HASH_NALGOS] = {
      		.update_fn = git_hash_unknown_update,
      		.final_fn = git_hash_unknown_final,
      		.final_oid_fn = git_hash_unknown_final_oid,
    -+		.fast_init_fn = git_hash_unknown_init,
    -+		.fast_clone_fn = git_hash_unknown_clone,
    -+		.fast_update_fn = git_hash_unknown_update,
    -+		.fast_final_fn = git_hash_unknown_final,
    -+		.fast_final_oid_fn = git_hash_unknown_final_oid,
    ++		.unsafe_init_fn = git_hash_unknown_init,
    ++		.unsafe_clone_fn = git_hash_unknown_clone,
    ++		.unsafe_update_fn = git_hash_unknown_update,
    ++		.unsafe_final_fn = git_hash_unknown_final,
    ++		.unsafe_final_oid_fn = git_hash_unknown_final_oid,
      		.empty_tree = NULL,
      		.empty_blob = NULL,
      		.null_oid = NULL,
    @@ object-file.c: const struct git_hash_algo hash_algos[GIT_HASH_NALGOS] = {
      		.update_fn = git_hash_sha1_update,
      		.final_fn = git_hash_sha1_final,
      		.final_oid_fn = git_hash_sha1_final_oid,
    -+		.fast_init_fn = git_hash_sha1_init_fast,
    -+		.fast_clone_fn = git_hash_sha1_clone_fast,
    -+		.fast_update_fn = git_hash_sha1_update_fast,
    -+		.fast_final_fn = git_hash_sha1_final_fast,
    -+		.fast_final_oid_fn = git_hash_sha1_final_oid_fast,
    ++		.unsafe_init_fn = git_hash_sha1_init_unsafe,
    ++		.unsafe_clone_fn = git_hash_sha1_clone_unsafe,
    ++		.unsafe_update_fn = git_hash_sha1_update_unsafe,
    ++		.unsafe_final_fn = git_hash_sha1_final_unsafe,
    ++		.unsafe_final_oid_fn = git_hash_sha1_final_oid_unsafe,
      		.empty_tree = &empty_tree_oid,
      		.empty_blob = &empty_blob_oid,
      		.null_oid = &null_oid_sha1,
    @@ object-file.c: const struct git_hash_algo hash_algos[GIT_HASH_NALGOS] = {
      		.update_fn = git_hash_sha256_update,
      		.final_fn = git_hash_sha256_final,
      		.final_oid_fn = git_hash_sha256_final_oid,
    -+		.fast_init_fn = git_hash_sha256_init,
    -+		.fast_clone_fn = git_hash_sha256_clone,
    -+		.fast_update_fn = git_hash_sha256_update,
    -+		.fast_final_fn = git_hash_sha256_final,
    -+		.fast_final_oid_fn = git_hash_sha256_final_oid,
    ++		.unsafe_init_fn = git_hash_sha256_init,
    ++		.unsafe_clone_fn = git_hash_sha256_clone,
    ++		.unsafe_update_fn = git_hash_sha256_update,
    ++		.unsafe_final_fn = git_hash_sha256_final,
    ++		.unsafe_final_oid_fn = git_hash_sha256_final_oid,
      		.empty_tree = &empty_tree_oid_sha256,
      		.empty_blob = &empty_blob_oid_sha256,
      		.null_oid = &null_oid_sha256,
 8:  137ec30d68a !  7:  af8fd9aa4ed Makefile: allow specifying a SHA-1 for non-cryptographic uses
    @@ Metadata
      ## Commit message ##
         Makefile: allow specifying a SHA-1 for non-cryptographic uses
     
    -    Introduce _FAST variants of the OPENSSL_SHA1, BLK_SHA1, and
    +    Introduce _UNSAFE variants of the OPENSSL_SHA1, BLK_SHA1, and
         APPLE_COMMON_CRYPTO_SHA1 compile-time knobs which indicate which SHA-1
         implementation is to be used for non-cryptographic uses.
     
    @@ Makefile: include shared.mak
      # Define APPLE_COMMON_CRYPTO_SHA1 to use Apple's CommonCrypto for
      # SHA-1.
      #
    -+# Define the same Makefile knobs as above, but suffixed with _FAST to
    -+# use the corresponding implementations for "fast" SHA-1 hashing for
    ++# Define the same Makefile knobs as above, but suffixed with _UNSAFE to
    ++# use the corresponding implementations for unsafe SHA-1 hashing for
     +# non-cryptographic purposes.
     +#
      # If don't enable any of the *_SHA1 settings in this section, Git will
    @@ Makefile: endif
      endif
      endif
      
    -+ifdef OPENSSL_SHA1_FAST
    ++ifdef OPENSSL_SHA1_UNSAFE
     +ifndef OPENSSL_SHA1
     +	EXTLIBS += $(LIB_4_CRYPTO)
    -+	BASIC_CFLAGS += -DSHA1_OPENSSL_FAST
    ++	BASIC_CFLAGS += -DSHA1_OPENSSL_UNSAFE
     +endif
     +else
    -+ifdef BLK_SHA1_FAST
    ++ifdef BLK_SHA1_UNSAFE
     +ifndef BLK_SHA1
     +	LIB_OBJS += block-sha1/sha1.o
    -+	BASIC_CFLAGS += -DSHA1_BLK_FAST
    ++	BASIC_CFLAGS += -DSHA1_BLK_UNSAFE
     +endif
     +else
    -+ifdef APPLE_COMMON_CRYPTO_SHA1_FAST
    ++ifdef APPLE_COMMON_CRYPTO_SHA1_UNSAFE
     +ifndef APPLE_COMMON_CRYPTO_SHA1
     +	COMPAT_CFLAGS += -DCOMMON_DIGEST_FOR_OPENSSL
    -+	BASIC_CFLAGS += -DSHA1_APPLE_FAST
    ++	BASIC_CFLAGS += -DSHA1_APPLE_UNSAFE
     +endif
     +endif
     +endif
    @@ hash.h
      #include "block-sha1/sha1.h"
      #endif
      
    -+#if defined(SHA1_APPLE_FAST)
    ++#if defined(SHA1_APPLE_UNSAFE)
     +#  include <CommonCrypto/CommonDigest.h>
    -+#  define platform_SHA_CTX_fast CC_SHA1_CTX
    -+#  define platform_SHA1_Init_fast CC_SHA1_Init
    -+#  define platform_SHA1_Update_fast CC_SHA1_Update
    -+#  define platform_SHA1_Final_fast CC_SHA1_Final
    -+#elif defined(SHA1_OPENSSL_FAST)
    ++#  define platform_SHA_CTX_unsafe CC_SHA1_CTX
    ++#  define platform_SHA1_Init_unsafe CC_SHA1_Init
    ++#  define platform_SHA1_Update_unsafe CC_SHA1_Update
    ++#  define platform_SHA1_Final_unsafe CC_SHA1_Final
    ++#elif defined(SHA1_OPENSSL_UNSAFE)
     +#  include <openssl/sha.h>
     +#  if defined(OPENSSL_API_LEVEL) && OPENSSL_API_LEVEL >= 3
    -+#    define SHA1_NEEDS_CLONE_HELPER_FAST
    ++#    define SHA1_NEEDS_CLONE_HELPER_UNSAFE
     +#    include "sha1/openssl.h"
    -+#    define platform_SHA_CTX_fast openssl_SHA1_CTX
    -+#    define platform_SHA1_Init_fast openssl_SHA1_Init
    -+#    define platform_SHA1_Clone_fast openssl_SHA1_Clone
    -+#    define platform_SHA1_Update_fast openssl_SHA1_Update
    -+#    define platform_SHA1_Final_fast openssl_SHA1_Final
    ++#    define platform_SHA_CTX_unsafe openssl_SHA1_CTX
    ++#    define platform_SHA1_Init_unsafe openssl_SHA1_Init
    ++#    define platform_SHA1_Clone_unsafe openssl_SHA1_Clone
    ++#    define platform_SHA1_Update_unsafe openssl_SHA1_Update
    ++#    define platform_SHA1_Final_unsafe openssl_SHA1_Final
     +#  else
    -+#    define platform_SHA_CTX_fast SHA_CTX
    -+#    define platform_SHA1_Init_fast SHA1_Init
    -+#    define platform_SHA1_Update_fast SHA1_Update
    -+#    define platform_SHA1_Final_fast SHA1_Final
    ++#    define platform_SHA_CTX_unsafe SHA_CTX
    ++#    define platform_SHA1_Init_unsafe SHA1_Init
    ++#    define platform_SHA1_Update_unsafe SHA1_Update
    ++#    define platform_SHA1_Final_unsafe SHA1_Final
     +#  endif
    -+#elif defined(SHA1_BLK_FAST)
    ++#elif defined(SHA1_BLK_UNSAFE)
     +#  include "block-sha1/sha1.h"
    -+#  define platform_SHA_CTX_fast blk_SHA_CTX
    -+#  define platform_SHA1_Init_fast blk_SHA1_Init
    -+#  define platform_SHA1_Update_fast blk_SHA1_Update
    -+#  define platform_SHA1_Final_fast blk_SHA1_Final
    ++#  define platform_SHA_CTX_unsafe blk_SHA_CTX
    ++#  define platform_SHA1_Init_unsafe blk_SHA1_Init
    ++#  define platform_SHA1_Update_unsafe blk_SHA1_Update
    ++#  define platform_SHA1_Final_unsafe blk_SHA1_Final
     +#endif
     +
      #if defined(SHA256_NETTLE)
 9:  4018261366f !  8:  4b83dd05e9f csum-file.c: use fast SHA-1 implementation when available
    @@ Metadata
     Author: Taylor Blau <me@ttaylorr.com>
     
      ## Commit message ##
    -    csum-file.c: use fast SHA-1 implementation when available
    +    csum-file.c: use unsafe SHA-1 implementation when available
     
    -    Update hashwrite() and friends to use the fast_-variants of hashing
    -    functions, calling for e.g., "the_hash_algo->fast_update_fn()" instead
    +    Update hashwrite() and friends to use the unsafe_-variants of hashing
    +    functions, calling for e.g., "the_hash_algo->unsafe_update_fn()" instead
         of "the_hash_algo->update_fn()".
     
         These callers only use the_hash_algo to produce a checksum, which we
         depend on for data integrity, but not for cryptographic purposes, so
    -    these callers are safe to use the fast (and potentially non-collision
    +    these callers are safe to use the unsafe (and potentially non-collision
         detecting) SHA-1 implementation.
     
         To time this, I took a freshly packed copy of linux.git, and ran the
    -    following with and without the OPENSSL_SHA1_FAST=1 build-knob. Both
    +    following with and without the OPENSSL_SHA1_UNSAFE=1 build-knob. Both
         versions were compiled with -O3:
     
             $ git for-each-ref --format='%(objectname)' refs/heads refs/tags >in
             $ valgrind --tool=callgrind ~/src/git/git-pack-objects \
                 --revs --stdout --all-progress --use-bitmap-index <in >/dev/null
     
    -    Without OPENSSL_SHA1_FAST=1 (that is, using the collision-detecting
    +    Without OPENSSL_SHA1_UNSAFE=1 (that is, using the collision-detecting
         SHA-1 implementation for both cryptographic and non-cryptographic
         purposes), we spend a significant amount of our instruction count in
         hashwrite():
    @@ Commit message
         , and the resulting "clone" takes 19.219 seconds of wall clock time,
         18.94 seconds of user time and 0.28 seconds of system time.
     
    -    Compiling with OPENSSL_SHA1_FAST=1, we spend ~60% fewer instructions in
    -    hashwrite():
    +    Compiling with OPENSSL_SHA1_UNSAFE=1, we spend ~60% fewer instructions
    +    in hashwrite():
     
             $ callgrind_annotate --inclusive=yes | grep hashwrite | head -n1
              59,164,001,176 (58.79%)  /home/ttaylorr/src/git/csum-file.c:hashwrite [/home/ttaylorr/src/git/git-pack-objects]
     
    -    , and generate the resulting "clone" much faster, in only 11.597 seconds
    +    , and generate the resulting "clone" much unsafeer, in only 11.597 seconds
         of wall time, 11.37 seconds of user time, and 0.23 seconds of system
         time, for a ~40% speed-up.
     
    @@ csum-file.c: void hashflush(struct hashfile *f)
      	if (offset) {
      		if (!f->skip_hash)
     -			the_hash_algo->update_fn(&f->ctx, f->buffer, offset);
    -+			the_hash_algo->fast_update_fn(&f->ctx, f->buffer, offset);
    ++			the_hash_algo->unsafe_update_fn(&f->ctx, f->buffer, offset);
      		flush(f, f->buffer, offset);
      		f->offset = 0;
      	}
    @@ csum-file.c: int finalize_hashfile(struct hashfile *f, unsigned char *result,
      		hashclr(f->buffer, the_repository->hash_algo);
      	else
     -		the_hash_algo->final_fn(f->buffer, &f->ctx);
    -+		the_hash_algo->fast_final_fn(f->buffer, &f->ctx);
    ++		the_hash_algo->unsafe_final_fn(f->buffer, &f->ctx);
      
      	if (result)
      		hashcpy(result, f->buffer, the_repository->hash_algo);
    @@ csum-file.c: void hashwrite(struct hashfile *f, const void *buf, unsigned int co
      			 */
      			if (!f->skip_hash)
     -				the_hash_algo->update_fn(&f->ctx, buf, nr);
    -+				the_hash_algo->fast_update_fn(&f->ctx, buf, nr);
    ++				the_hash_algo->unsafe_update_fn(&f->ctx, buf, nr);
      			flush(f, buf, nr);
      		} else {
      			/*
    @@ csum-file.c: static struct hashfile *hashfd_internal(int fd, const char *name,
      	f->do_crc = 0;
      	f->skip_hash = 0;
     -	the_hash_algo->init_fn(&f->ctx);
    -+	the_hash_algo->fast_init_fn(&f->ctx);
    ++	the_hash_algo->unsafe_init_fn(&f->ctx);
      
      	f->buffer_len = buffer_len;
      	f->buffer = xmalloc(buffer_len);
    @@ csum-file.c: void hashfile_checkpoint(struct hashfile *f, struct hashfile_checkp
      	hashflush(f);
      	checkpoint->offset = f->total;
     -	the_hash_algo->clone_fn(&checkpoint->ctx, &f->ctx);
    -+	the_hash_algo->fast_clone_fn(&checkpoint->ctx, &f->ctx);
    ++	the_hash_algo->unsafe_clone_fn(&checkpoint->ctx, &f->ctx);
      }
      
      int hashfile_truncate(struct hashfile *f, struct hashfile_checkpoint *checkpoint)
    @@ csum-file.c: int hashfile_truncate(struct hashfile *f, struct hashfile_checkpoin
      		return -1;
      	f->total = offset;
     -	the_hash_algo->clone_fn(&f->ctx, &checkpoint->ctx);
    -+	the_hash_algo->fast_clone_fn(&f->ctx, &checkpoint->ctx);
    ++	the_hash_algo->unsafe_clone_fn(&f->ctx, &checkpoint->ctx);
      	f->offset = 0; /* hashflush() was called in checkpoint */
      	return 0;
      }
    @@ csum-file.c: int hashfile_checksum_valid(const unsigned char *data, size_t total
     -	the_hash_algo->init_fn(&ctx);
     -	the_hash_algo->update_fn(&ctx, data, data_len);
     -	the_hash_algo->final_fn(got, &ctx);
    -+	the_hash_algo->fast_init_fn(&ctx);
    -+	the_hash_algo->fast_update_fn(&ctx, data, data_len);
    -+	the_hash_algo->fast_final_fn(got, &ctx);
    ++	the_hash_algo->unsafe_init_fn(&ctx);
    ++	the_hash_algo->unsafe_update_fn(&ctx, data, data_len);
    ++	the_hash_algo->unsafe_final_fn(got, &ctx);
      
      	return hasheq(got, data + data_len, the_repository->hash_algo);
      }

base-commit: 6258f68c3c1092c901337895c864073dcdea9213
-- 
2.46.2.636.g4b83dd05e9f

[PATCH v4 1/8] finalize_object_file(): check for name collision before renaming

[Taylor Blau]

We prefer link()/unlink() to rename() for object files, with the idea
that we should prefer the data that is already on disk to what is
incoming. But we may fall back to rename() if the user has configured us
to do so, or if the filesystem seems not to support cross-directory
links. This loses the "prefer what is on disk" property.

We can mitigate this somewhat by trying to stat() the destination
filename before doing the rename. This is racy, since the object could
be created between the stat() and rename() calls. But in practice it is
expanding the definition of "what is already on disk" to be the point
that the function is called. That is enough to deal with any potential
attacks where an attacker is trying to collide hashes with what's
already in the repository.

Co-authored-by: Jeff King <peff@peff.net>
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Taylor Blau <me@ttaylorr.com>
---
 object-file.c | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/object-file.c b/object-file.c
index 968da27cd41..38407f468a9 100644
--- a/object-file.c
+++ b/object-file.c
@@ -1937,6 +1937,7 @@ static void write_object_file_prepare_literally(const struct git_hash_algo *algo
  */
 int finalize_object_file(const char *tmpfile, const char *filename)
 {
+	struct stat st;
 	int ret = 0;
 
 	if (object_creation_mode == OBJECT_CREATION_USES_RENAMES)
@@ -1957,9 +1958,12 @@ int finalize_object_file(const char *tmpfile, const char *filename)
 	 */
 	if (ret && ret != EEXIST) {
 	try_rename:
-		if (!rename(tmpfile, filename))
+		if (!stat(filename, &st))
+			ret = EEXIST;
+		else if (!rename(tmpfile, filename))
 			goto out;
-		ret = errno;
+		else
+			ret = errno;
 	}
 	unlink_or_warn(tmpfile);
 	if (ret) {
-- 
2.46.2.636.g4b83dd05e9f

Re: [PATCH v4 1/8] finalize_object_file(): check for name collision before renaming

[Junio C Hamano]

Taylor Blau <me@ttaylorr.com> writes:

> ... But in practice it is
> expanding the definition of "what is already on disk" to be the point
> that the function is called.

Yeah, it is a reasonable argument for this additional protection.
It does not make things worse.  All it takes is for the attacker to
come a bit earlier to defeat the link/unlink dance, so doing it "the
right way" does not make it fundamentally safer.

I hope all TOCTOU races can be explained away this way ;-).

> Co-authored-by: Jeff King <peff@peff.net>
> Signed-off-by: Jeff King <peff@peff.net>
> Signed-off-by: Taylor Blau <me@ttaylorr.com>
> ---
>  object-file.c | 8 ++++++--
>  1 file changed, 6 insertions(+), 2 deletions(-)
>
> diff --git a/object-file.c b/object-file.c
> index 968da27cd41..38407f468a9 100644
> --- a/object-file.c
> +++ b/object-file.c
> @@ -1937,6 +1937,7 @@ static void write_object_file_prepare_literally(const struct git_hash_algo *algo
>   */
>  int finalize_object_file(const char *tmpfile, const char *filename)
>  {
> +	struct stat st;
>  	int ret = 0;
>  
>  	if (object_creation_mode == OBJECT_CREATION_USES_RENAMES)
> @@ -1957,9 +1958,12 @@ int finalize_object_file(const char *tmpfile, const char *filename)
>  	 */
>  	if (ret && ret != EEXIST) {
>  	try_rename:
> -		if (!rename(tmpfile, filename))
> +		if (!stat(filename, &st))
> +			ret = EEXIST;
> +		else if (!rename(tmpfile, filename))
>  			goto out;
> -		ret = errno;
> +		else
> +			ret = errno;
>  	}
>  	unlink_or_warn(tmpfile);
>  	if (ret) {

[PATCH v4 2/8] finalize_object_file(): refactor unlink_or_warn() placement

[Taylor Blau]

As soon as we've tried to link() a temporary object into place, we then
unlink() the tempfile immediately, whether we were successful or not.

For the success case, this is because we no longer need the old file
(it's now linked into place).

For the error case, there are two outcomes. Either we got EEXIST, in
which case we consider the collision to be a noop. Or we got a system
error, in which we case we are just cleaning up after ourselves.

Using a single line for all of these cases has some problems:

  - in the error case, our unlink() may clobber errno, which we use in
    the error message

  - for the collision case, there's a FIXME that indicates we should do
    a collision check. In preparation for implementing that, we'll need
    to actually hold on to the file.

Split these three cases into their own calls to unlink_or_warn(). This
is more verbose, but lets us do the right thing in each case.

Co-authored-by: Jeff King <peff@peff.net>
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Taylor Blau <me@ttaylorr.com>
---
 object-file.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/object-file.c b/object-file.c
index 38407f468a9..5a1da577115 100644
--- a/object-file.c
+++ b/object-file.c
@@ -1944,6 +1944,8 @@ int finalize_object_file(const char *tmpfile, const char *filename)
 		goto try_rename;
 	else if (link(tmpfile, filename))
 		ret = errno;
+	else
+		unlink_or_warn(tmpfile);
 
 	/*
 	 * Coda hack - coda doesn't like cross-directory links,
@@ -1965,12 +1967,15 @@ int finalize_object_file(const char *tmpfile, const char *filename)
 		else
 			ret = errno;
 	}
-	unlink_or_warn(tmpfile);
 	if (ret) {
 		if (ret != EEXIST) {
+			int saved_errno = errno;
+			unlink_or_warn(tmpfile);
+			errno = saved_errno;
 			return error_errno(_("unable to write file %s"), filename);
 		}
 		/* FIXME!!! Collision check here ? */
+		unlink_or_warn(tmpfile);
 	}
 
 out:
-- 
2.46.2.636.g4b83dd05e9f

[PATCH v4 3/8] finalize_object_file(): implement collision check

[Taylor Blau]

We've had "FIXME!!! Collision check here ?" in finalize_object_file()
since aac1794132 (Improve sha1 object file writing., 2005-05-03). That
is, when we try to write a file with the same name, we assume the
on-disk contents are the same and blindly throw away the new copy.

One of the reasons we never implemented this is because the files it
moves are all named after the cryptographic hash of their contents
(either loose objects, or packs which have their hash in the name these
days). So we are unlikely to see such a collision by accident. And even
though there are weaknesses in sha1, we assume they are mitigated by our
use of sha1dc.

So while it's a theoretical concern now, it hasn't been a priority.
However, if we start using weaker hashes for pack checksums and names,
this will become a practical concern. So in preparation, let's actually
implement a byte-for-byte collision check.

The new check will cause the write of new differing content to be a
failure, rather than a silent noop, and we'll retain the temporary file
on disk. If there's no collision present, we'll clean up the temporary
file as usual after either rename()-ing or link()-ing it into place.

Note that this may cause some extra computation when the files are in
fact identical, but this should happen rarely.

Loose objects are exempt from this check, and the collision check may be
skipped by calling the _flags variant of this function with the
FOF_SKIP_COLLISION_CHECK bit set. This is done for a couple of reasons:

  - We don't treat the hash of the loose object file's contents as a
    checksum, since the same loose object can be stored using different
    bytes on disk (e.g., when adjusting core.compression, using a
    different version of zlib, etc.).

    This is fundamentally different from cases where
    finalize_object_file() is operating over a file which uses the hash
    value as a checksum of the contents. In other words, a pair of
    identical loose objects can be stored using different bytes on disk,
    and that should not be treated as a collision.

  - We already use the path of the loose object as its hash value /
    object name, so checking for collisions at the content level doesn't
    add anything.

    This is why we do not bother to check the inflated object contents
    for collisions either, since either (a) the object contents have the
    fingerprint of a SHA-1 collision, in which case the collision
    detecting SHA-1 implementation used to hash the contents to give us
    a path would have already rejected it, or (b) the contents are part
    of a colliding pair which does not bear the same fingerprints of
    known collision attacks, in which case we would not have caught it
    anyway.

    So skipping the collision check here does not change for better or
    worse the hardness of loose object writes.

As a small note related to the latter bullet point above, we must teach
the tmp-objdir routines to similarly skip the content-level collision
checks when calling migrate_one() on a loose object file, which we do by
setting the FOF_SKIP_COLLISION_CHECK bit when we are inside of a loose
object shard.

Co-authored-by: Jeff King <peff@peff.net>
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Taylor Blau <me@ttaylorr.com>
---
 object-file.c | 67 ++++++++++++++++++++++++++++++++++++++++++++++++---
 object-file.h |  6 +++++
 tmp-objdir.c  | 26 ++++++++++++++------
 3 files changed, 89 insertions(+), 10 deletions(-)

diff --git a/object-file.c b/object-file.c
index 5a1da577115..b9a3a1f62db 100644
--- a/object-file.c
+++ b/object-file.c
@@ -1932,10 +1932,67 @@ static void write_object_file_prepare_literally(const struct git_hash_algo *algo
 	hash_object_body(algo, &c, buf, len, oid, hdr, hdrlen);
 }
 
+static int check_collision(const char *filename_a, const char *filename_b)
+{
+	char buf_a[4096], buf_b[4096];
+	int fd_a = -1, fd_b = -1;
+	int ret = 0;
+
+	fd_a = open(filename_a, O_RDONLY);
+	if (fd_a < 0) {
+		ret = error_errno(_("unable to open %s"), filename_a);
+		goto out;
+	}
+
+	fd_b = open(filename_b, O_RDONLY);
+	if (fd_b < 0) {
+		ret = error_errno(_("unable to open %s"), filename_b);
+		goto out;
+	}
+
+	while (1) {
+		ssize_t sz_a, sz_b;
+
+		sz_a = read_in_full(fd_a, buf_a, sizeof(buf_a));
+		if (sz_a < 0) {
+			ret = error_errno(_("unable to read %s"), filename_a);
+			goto out;
+		}
+
+		sz_b = read_in_full(fd_b, buf_b, sizeof(buf_b));
+		if (sz_b < 0) {
+			ret = error_errno(_("unable to read %s"), filename_b);
+			goto out;
+		}
+
+		if (sz_a != sz_b || memcmp(buf_a, buf_b, sz_a)) {
+			ret = error(_("files '%s' and '%s' differ in contents"),
+				    filename_a, filename_b);
+			goto out;
+		}
+
+		if (sz_a < sizeof(buf_a))
+			break;
+	}
+
+out:
+	if (fd_a > -1)
+		close(fd_a);
+	if (fd_b > -1)
+		close(fd_b);
+	return ret;
+}
+
 /*
  * Move the just written object into its final resting place.
  */
 int finalize_object_file(const char *tmpfile, const char *filename)
+{
+	return finalize_object_file_flags(tmpfile, filename, 0);
+}
+
+int finalize_object_file_flags(const char *tmpfile, const char *filename,
+			       enum finalize_object_file_flags flags)
 {
 	struct stat st;
 	int ret = 0;
@@ -1974,7 +2031,9 @@ int finalize_object_file(const char *tmpfile, const char *filename)
 			errno = saved_errno;
 			return error_errno(_("unable to write file %s"), filename);
 		}
-		/* FIXME!!! Collision check here ? */
+		if (!(flags & FOF_SKIP_COLLISION_CHECK) &&
+		    check_collision(tmpfile, filename))
+				return -1;
 		unlink_or_warn(tmpfile);
 	}
 
@@ -2228,7 +2287,8 @@ static int write_loose_object(const struct object_id *oid, char *hdr,
 			warning_errno(_("failed utime() on %s"), tmp_file.buf);
 	}
 
-	return finalize_object_file(tmp_file.buf, filename.buf);
+	return finalize_object_file_flags(tmp_file.buf, filename.buf,
+					  FOF_SKIP_COLLISION_CHECK);
 }
 
 static int freshen_loose_object(const struct object_id *oid)
@@ -2350,7 +2410,8 @@ int stream_loose_object(struct input_stream *in_stream, size_t len,
 		strbuf_release(&dir);
 	}
 
-	err = finalize_object_file(tmp_file.buf, filename.buf);
+	err = finalize_object_file_flags(tmp_file.buf, filename.buf,
+					 FOF_SKIP_COLLISION_CHECK);
 	if (!err && compat)
 		err = repo_add_loose_object_map(the_repository, oid, &compat_oid);
 cleanup:
diff --git a/object-file.h b/object-file.h
index d6414610f80..81b30d269c8 100644
--- a/object-file.h
+++ b/object-file.h
@@ -117,7 +117,13 @@ int check_object_signature(struct repository *r, const struct object_id *oid,
  */
 int stream_object_signature(struct repository *r, const struct object_id *oid);
 
+enum finalize_object_file_flags {
+	FOF_SKIP_COLLISION_CHECK = 1,
+};
+
 int finalize_object_file(const char *tmpfile, const char *filename);
+int finalize_object_file_flags(const char *tmpfile, const char *filename,
+			       enum finalize_object_file_flags flags);
 
 /* Helper to check and "touch" a file */
 int check_and_freshen_file(const char *fn, int freshen);
diff --git a/tmp-objdir.c b/tmp-objdir.c
index c2fb9f91930..9da0071cba8 100644
--- a/tmp-objdir.c
+++ b/tmp-objdir.c
@@ -206,9 +206,11 @@ static int read_dir_paths(struct string_list *out, const char *path)
 	return 0;
 }
 
-static int migrate_paths(struct strbuf *src, struct strbuf *dst);
+static int migrate_paths(struct strbuf *src, struct strbuf *dst,
+			 enum finalize_object_file_flags flags);
 
-static int migrate_one(struct strbuf *src, struct strbuf *dst)
+static int migrate_one(struct strbuf *src, struct strbuf *dst,
+		       enum finalize_object_file_flags flags)
 {
 	struct stat st;
 
@@ -220,12 +222,18 @@ static int migrate_one(struct strbuf *src, struct strbuf *dst)
 				return -1;
 		} else if (errno != EEXIST)
 			return -1;
-		return migrate_paths(src, dst);
+		return migrate_paths(src, dst, flags);
 	}
-	return finalize_object_file(src->buf, dst->buf);
+	return finalize_object_file_flags(src->buf, dst->buf, flags);
 }
 
-static int migrate_paths(struct strbuf *src, struct strbuf *dst)
+static int is_loose_object_shard(const char *name)
+{
+	return strlen(name) == 2 && isxdigit(name[0]) && isxdigit(name[1]);
+}
+
+static int migrate_paths(struct strbuf *src, struct strbuf *dst,
+			 enum finalize_object_file_flags flags)
 {
 	size_t src_len = src->len, dst_len = dst->len;
 	struct string_list paths = STRING_LIST_INIT_DUP;
@@ -239,11 +247,15 @@ static int migrate_paths(struct strbuf *src, struct strbuf *dst)
 
 	for (i = 0; i < paths.nr; i++) {
 		const char *name = paths.items[i].string;
+		enum finalize_object_file_flags flags_copy = flags;
 
 		strbuf_addf(src, "/%s", name);
 		strbuf_addf(dst, "/%s", name);
 
-		ret |= migrate_one(src, dst);
+		if (is_loose_object_shard(name))
+			flags_copy |= FOF_SKIP_COLLISION_CHECK;
+
+		ret |= migrate_one(src, dst, flags_copy);
 
 		strbuf_setlen(src, src_len);
 		strbuf_setlen(dst, dst_len);
@@ -271,7 +283,7 @@ int tmp_objdir_migrate(struct tmp_objdir *t)
 	strbuf_addbuf(&src, &t->path);
 	strbuf_addstr(&dst, repo_get_object_directory(the_repository));
 
-	ret = migrate_paths(&src, &dst);
+	ret = migrate_paths(&src, &dst, 0);
 
 	strbuf_release(&src);
 	strbuf_release(&dst);
-- 
2.46.2.636.g4b83dd05e9f

Re: [PATCH v4 3/8] finalize_object_file(): implement collision check

[Jeff King]

On Tue, Sep 24, 2024 at 01:32:17PM -0400, Taylor Blau wrote:

> Loose objects are exempt from this check, and the collision check may be
> skipped by calling the _flags variant of this function with the
> FOF_SKIP_COLLISION_CHECK bit set. This is done for a couple of reasons:
> 
>   - We don't treat the hash of the loose object file's contents as a
>     checksum, since the same loose object can be stored using different
>     bytes on disk (e.g., when adjusting core.compression, using a
>     different version of zlib, etc.).
> 
>     This is fundamentally different from cases where
>     finalize_object_file() is operating over a file which uses the hash
>     value as a checksum of the contents. In other words, a pair of
>     identical loose objects can be stored using different bytes on disk,
>     and that should not be treated as a collision.

OK, this explains why a byte-for-byte hash-check would be the wrong
thing (it would cause false positives)...

>   - We already use the path of the loose object as its hash value /
>     object name, so checking for collisions at the content level doesn't
>     add anything.
> 
>     This is why we do not bother to check the inflated object contents
>     for collisions either, since either (a) the object contents have the
>     fingerprint of a SHA-1 collision, in which case the collision
>     detecting SHA-1 implementation used to hash the contents to give us
>     a path would have already rejected it, or (b) the contents are part
>     of a colliding pair which does not bear the same fingerprints of
>     known collision attacks, in which case we would not have caught it
>     anyway.
> 
>     So skipping the collision check here does not change for better or
>     worse the hardness of loose object writes.

...and this is the argument for why it is not necessary to do a
content-level check. As you say, we're not making anything worse here,
as we've never implemented this collision check. But I think the "FIXME"
in the code was really there under the notion that it was a backstop to
SHA-1 being broken (belt-and-suspenders, if you will). And your argument
above assumes that SHA-1 (using the DC variant) is safe.

But I think we can expand the argument a bit. I don't think this is a
very good place for such a collision check, because race conditions
aside, we'll already have bailed long before! When we do a write via
write_object_file(), for example, we'll hit the freshen paths that
assume the contents are identical, and skip the write (and thus this
finalize) entirely.

So if you want a collision check, we have to do it at a much higher
level. And indeed we do: index-pack already implements such a check
(through the confusingly named "check_collison" function). I don't think
unpack-objects does so (it looks like it just feeds the result to
write_object_file(), which does the freshening thing).

So the argument I'd make here is more like: this is the wrong place to
do it.

  Side thoughts on collision checks:

    I think index-pack is safe in the sense that it will always prefer
    on-disk copies and will complain if it sees a collision.
    unpack-objects is not, nor are regular in-repo writes (which
    normally cannot be triggered by remote, but on forges that do
    merges, etc, that's not always true). Both of the latter are also
    subject to races, where a simultaneous collision might let the
    attacker win.

    That race is kind of moot in a world where anybody can push to a
    fork of a repo that ends up in the same shared location (so they can
    actually win and become the "on disk" copy), and we're relying on
    sha1dc for protection there. That's specific to certain forges, but
    they do represent a lot of Git use.

    In general, the use of unpack-objects versus index-pack is up to the
    attacker (based on pack size). So I think it would be nice if
    unpack-objects did the collision check. Even if the attacker beats
    you to writing the object, it would be nice to see "holy crap, there
    was a collision" instead of just silently throwing your pushed
    object away.

    I know that GitHub only ever runs index-pack, which may give some
    amount of protection here. In general, I think we should consider
    deprecating unpack-objects in favor of teaching index-pack to
    unpack. It has many enhancements (this one, but also threading) that
    unpack-objects does not. I have an old patch series for this (sorry,
    only from 2017, I'm slipping). IIRC the sticking point was that
    unpack-objects is better about memory use in some cases (streaming
    large blobs?) and I hadn't figured out a way around that.

Phew. Sorry, that was a long digression for "I think you're right, but I
might have argued it a little differently". I think the direction of the
patch (skipping checks entirely for loose objects) is the right thing.

> As a small note related to the latter bullet point above, we must teach
> the tmp-objdir routines to similarly skip the content-level collision
> checks when calling migrate_one() on a loose object file, which we do by
> setting the FOF_SKIP_COLLISION_CHECK bit when we are inside of a loose
> object shard.

OK, this is the part I was wondering how you were going to deal with. :)
Let's look at the implementation...

> @@ -239,11 +247,15 @@ static int migrate_paths(struct strbuf *src, struct strbuf *dst)
>  
>  	for (i = 0; i < paths.nr; i++) {
>  		const char *name = paths.items[i].string;
> +		enum finalize_object_file_flags flags_copy = flags;
>  
>  		strbuf_addf(src, "/%s", name);
>  		strbuf_addf(dst, "/%s", name);
>  
> -		ret |= migrate_one(src, dst);
> +		if (is_loose_object_shard(name))
> +			flags_copy |= FOF_SKIP_COLLISION_CHECK;
> +
> +		ret |= migrate_one(src, dst, flags_copy);
>  
>  		strbuf_setlen(src, src_len);
>  		strbuf_setlen(dst, dst_len);

This looks pretty reasonable overall, though I'd note that
migrate_paths() is called recursively. So if I had:

  tmp-objdir-XXXXXX/pack/ab/foo.pack

we'd skip the collision check. I'm not sure how much we want to worry
about that. I don't think we'd ever create such a path, and the general
form of the paths is all under local control, so an attacker can't
convince us to do so. And I find it pretty unlikely that we'd change the
on-disk layout to accidentally trigger this.

So even though there are security implications if we have a false
positive from is_loose_object_shard(), I don't know that it's worth
caring about.

If we did want to, I think a more robust check is to make sure the shard
comes at the start of the path. Which is tricky, of course, because
we're building a path on top of an arbitrary root (tmp-objdir for
the src, and the repo object dir for the dst).

So you'd have to save that base_len and use it as a root, like the patch
below. I'm OK if you want to write all of this off as over-engineering,
though.

diff --git a/tmp-objdir.c b/tmp-objdir.c
index 9da0071cba..27ba9f4f57 100644
--- a/tmp-objdir.c
+++ b/tmp-objdir.c
@@ -207,10 +207,18 @@ static int read_dir_paths(struct string_list *out, const char *path)
 }
 
 static int migrate_paths(struct strbuf *src, struct strbuf *dst,
-			 enum finalize_object_file_flags flags);
+			 size_t base_len);
+
+static int is_loose_path(const char *name)
+{
+	return *name++ == '/' &&
+	       isxdigit(*name++) &&
+	       isxdigit(*name++) &&
+	       *name++ == '/';
+}
 
 static int migrate_one(struct strbuf *src, struct strbuf *dst,
-		       enum finalize_object_file_flags flags)
+		       size_t base_len)
 {
 	struct stat st;
 
@@ -222,18 +230,16 @@ static int migrate_one(struct strbuf *src, struct strbuf *dst,
 				return -1;
 		} else if (errno != EEXIST)
 			return -1;
-		return migrate_paths(src, dst, flags);
+		return migrate_paths(src, dst, base_len);
 	}
-	return finalize_object_file_flags(src->buf, dst->buf, flags);
+	return finalize_object_file_flags(src->buf, dst->buf,
+					  is_loose_path(src->buf + base_len) ?
+					  FOF_SKIP_COLLISION_CHECK : 0);
 }
 
-static int is_loose_object_shard(const char *name)
-{
-	return strlen(name) == 2 && isxdigit(name[0]) && isxdigit(name[1]);
-}
 
 static int migrate_paths(struct strbuf *src, struct strbuf *dst,
-			 enum finalize_object_file_flags flags)
+			 size_t base_len)
 {
 	size_t src_len = src->len, dst_len = dst->len;
 	struct string_list paths = STRING_LIST_INIT_DUP;
@@ -247,15 +253,11 @@ static int migrate_paths(struct strbuf *src, struct strbuf *dst,
 
 	for (i = 0; i < paths.nr; i++) {
 		const char *name = paths.items[i].string;
-		enum finalize_object_file_flags flags_copy = flags;
 
 		strbuf_addf(src, "/%s", name);
 		strbuf_addf(dst, "/%s", name);
 
-		if (is_loose_object_shard(name))
-			flags_copy |= FOF_SKIP_COLLISION_CHECK;
-
-		ret |= migrate_one(src, dst, flags_copy);
+		ret |= migrate_one(src, dst, base_len);
 
 		strbuf_setlen(src, src_len);
 		strbuf_setlen(dst, dst_len);
@@ -283,7 +285,7 @@ int tmp_objdir_migrate(struct tmp_objdir *t)
 	strbuf_addbuf(&src, &t->path);
 	strbuf_addstr(&dst, repo_get_object_directory(the_repository));
 
-	ret = migrate_paths(&src, &dst, 0);
+	ret = migrate_paths(&src, &dst, src.len);
 
 	strbuf_release(&src);
 	strbuf_release(&dst);

Re: [PATCH v4 3/8] finalize_object_file(): implement collision check

[Taylor Blau]

On Tue, Sep 24, 2024 at 04:37:18PM -0400, Jeff King wrote:
> But I think we can expand the argument a bit. I don't think this is a
> very good place for such a collision check, because race conditions
> aside, we'll already have bailed long before! When we do a write via
> write_object_file(), for example, we'll hit the freshen paths that
> assume the contents are identical, and skip the write (and thus this
> finalize) entirely.
>
> So if you want a collision check, we have to do it at a much higher
> level. And indeed we do: index-pack already implements such a check
> (through the confusingly named "check_collison" function). I don't think
> unpack-objects does so (it looks like it just feeds the result to
> write_object_file(), which does the freshening thing).
>
> So the argument I'd make here is more like: this is the wrong place to
> do it.

I think that is reasonable, and I agree with your reasoning here. I'm
happy to reword the commit message if you think doing so would be
useful, or drop the paragraph entirely if you think it's just confusing.

Let me know what you think, I'm happy to do whatever here, reroll or not
:-).

>   Side thoughts on collision checks:
>
>     I think index-pack is safe in the sense that it will always prefer
>     on-disk copies and will complain if it sees a collision.
>     unpack-objects is not, nor are regular in-repo writes (which
>     normally cannot be triggered by remote, but on forges that do
>     merges, etc, that's not always true). Both of the latter are also
>     subject to races, where a simultaneous collision might let the
>     attacker win.

Yup.

>     That race is kind of moot in a world where anybody can push to a
>     fork of a repo that ends up in the same shared location (so they can
>     actually win and become the "on disk" copy), and we're relying on
>     sha1dc for protection there. That's specific to certain forges, but
>     they do represent a lot of Git use.
>
>     In general, the use of unpack-objects versus index-pack is up to the
>     attacker (based on pack size). So I think it would be nice if
>     unpack-objects did the collision check. Even if the attacker beats
>     you to writing the object, it would be nice to see "holy crap, there
>     was a collision" instead of just silently throwing your pushed
>     object away.

Right, I agree that unpack-objects definitely should do the collision
check here. And indeed, that is the case, since that code (which all is
directly implemented in the builtin) uses the regular
collision-detecting SHA-1 implementation.

>     I know that GitHub only ever runs index-pack, which may give some
>     amount of protection here. In general, I think we should consider
>     deprecating unpack-objects in favor of teaching index-pack to
>     unpack. It has many enhancements (this one, but also threading) that
>     unpack-objects does not. I have an old patch series for this (sorry,
>     only from 2017, I'm slipping). IIRC the sticking point was that
>     unpack-objects is better about memory use in some cases (streaming
>     large blobs?) and I hadn't figured out a way around that.

Only seven years old? Sheesh, you really are slipping ;-).

> Phew. Sorry, that was a long digression for "I think you're right, but I
> might have argued it a little differently". I think the direction of the
> patch (skipping checks entirely for loose objects) is the right thing.
>
> > As a small note related to the latter bullet point above, we must teach
> > the tmp-objdir routines to similarly skip the content-level collision
> > checks when calling migrate_one() on a loose object file, which we do by
> > setting the FOF_SKIP_COLLISION_CHECK bit when we are inside of a loose
> > object shard.
>
> OK, this is the part I was wondering how you were going to deal with. :)
> Let's look at the implementation...
>
> > @@ -239,11 +247,15 @@ static int migrate_paths(struct strbuf *src, struct strbuf *dst)
> >
> >  	for (i = 0; i < paths.nr; i++) {
> >  		const char *name = paths.items[i].string;
> > +		enum finalize_object_file_flags flags_copy = flags;
> >
> >  		strbuf_addf(src, "/%s", name);
> >  		strbuf_addf(dst, "/%s", name);
> >
> > -		ret |= migrate_one(src, dst);
> > +		if (is_loose_object_shard(name))
> > +			flags_copy |= FOF_SKIP_COLLISION_CHECK;
> > +
> > +		ret |= migrate_one(src, dst, flags_copy);
> >
> >  		strbuf_setlen(src, src_len);
> >  		strbuf_setlen(dst, dst_len);
>
> This looks pretty reasonable overall, though I'd note that
> migrate_paths() is called recursively. So if I had:
>
>   tmp-objdir-XXXXXX/pack/ab/foo.pack
>
> we'd skip the collision check. I'm not sure how much we want to worry
> about that. I don't think we'd ever create such a path, and the general
> form of the paths is all under local control, so an attacker can't
> convince us to do so. And I find it pretty unlikely that we'd change the
> on-disk layout to accidentally trigger this.

I thought about this when writing it, and came to the conclusion that
the checks we have for "are we in something that looks like a loose
object shard?" are sane. That's only because we won't bother reading a
pack in $GIT_DIR/objects/pack/xx/yy.pack, since we do not read packs
recursively from the pack sub-directory.

So I think that the diff you posted below isn't hurting anything, and
the implementation looks correct to me, but I also don't think it's
helping anything either as a consequence of the above.

Thanks,
Taylor

Re: [PATCH v4 3/8] finalize_object_file(): implement collision check

[Jeff King]

On Tue, Sep 24, 2024 at 05:59:10PM -0400, Taylor Blau wrote:

> > So the argument I'd make here is more like: this is the wrong place to
> > do it.
> 
> I think that is reasonable, and I agree with your reasoning here. I'm
> happy to reword the commit message if you think doing so would be
> useful, or drop the paragraph entirely if you think it's just confusing.
> 
> Let me know what you think, I'm happy to do whatever here, reroll or not
> :-).

I'm content to let this live in the list archive, but it sounds like
Junio had the same reaction, so it may be worth trying to rework the
commit message a bit.

> >     In general, the use of unpack-objects versus index-pack is up to the
> >     attacker (based on pack size). So I think it would be nice if
> >     unpack-objects did the collision check. Even if the attacker beats
> >     you to writing the object, it would be nice to see "holy crap, there
> >     was a collision" instead of just silently throwing your pushed
> >     object away.
> 
> Right, I agree that unpack-objects definitely should do the collision
> check here. And indeed, that is the case, since that code (which all is
> directly implemented in the builtin) uses the regular
> collision-detecting SHA-1 implementation.

It uses sha1dc, but what I mean by "collision check" is an additional
belt-and-suspenders check. That even once we see an object which
hashes to a particular sha1 which we already have, we'll do the
byte-for-byte comparison. See index-pack's "check_collison".

And that's what I think should be added to unpack-objects (but again,
this is all orthogonal to your series).

> I thought about this when writing it, and came to the conclusion that
> the checks we have for "are we in something that looks like a loose
> object shard?" are sane. That's only because we won't bother reading a
> pack in $GIT_DIR/objects/pack/xx/yy.pack, since we do not read packs
> recursively from the pack sub-directory.
> 
> So I think that the diff you posted below isn't hurting anything, and
> the implementation looks correct to me, but I also don't think it's
> helping anything either as a consequence of the above.

Right, it's purely about future-proofing against a hypothetical change
where we'd be more inclusive (both in what we read, but also in what
we'd ourselves write!).

-Peff

Re: [PATCH v4 3/8] finalize_object_file(): implement collision check

[Taylor Blau]

On Tue, Sep 24, 2024 at 06:20:39PM -0400, Jeff King wrote:
> On Tue, Sep 24, 2024 at 05:59:10PM -0400, Taylor Blau wrote:
>
> > > So the argument I'd make here is more like: this is the wrong place to
> > > do it.
> >
> > I think that is reasonable, and I agree with your reasoning here. I'm
> > happy to reword the commit message if you think doing so would be
> > useful, or drop the paragraph entirely if you think it's just confusing.
> >
> > Let me know what you think, I'm happy to do whatever here, reroll or not
> > :-).
>
> I'm content to let this live in the list archive, but it sounds like
> Junio had the same reaction, so it may be worth trying to rework the
> commit message a bit.

Here's the relevant portion of my range-diff that has the new wording
(which is more or less equivalent to your "this isn't the right place to
do it, and we're not fundamentally changing anything from a security
perspective here" argument). Let me know what you think:

--- 8< ---
3:  ed9eeef851 ! 3:  41d38352a4 finalize_object_file(): implement collision check
    @@ Commit message
             object name, so checking for collisions at the content level doesn't
             add anything.

    -        This is why we do not bother to check the inflated object contents
    -        for collisions either, since either (a) the object contents have the
    -        fingerprint of a SHA-1 collision, in which case the collision
    -        detecting SHA-1 implementation used to hash the contents to give us
    -        a path would have already rejected it, or (b) the contents are part
    -        of a colliding pair which does not bear the same fingerprints of
    -        known collision attacks, in which case we would not have caught it
    -        anyway.
    +        Adding a content-level collision check would have to happen at a
    +        higher level than in finalize_object_file(), since (avoiding race
    +        conditions) writing an object loose which already exists in the
    +        repository will prevent us from even reaching finalize_object_file()
    +        via the object freshening code.
    +
    +        There is a collision check in index-pack via its `check_collision()`
    +        function, but there isn't an analogous function in unpack-objects,
    +        which just feeds the result to write_object_file().

             So skipping the collision check here does not change for better or
             worse the hardness of loose object writes.
    @@ Commit message

         Co-authored-by: Jeff King <peff@peff.net>
         Signed-off-by: Jeff King <peff@peff.net>
    +    Helped-by: Elijah Newren <newren@gmail.com>
         Signed-off-by: Taylor Blau <me@ttaylorr.com>

      ## object-file.c ##
--- >8 ---

Thanks,
Taylor

Re: [PATCH v4 3/8] finalize_object_file(): implement collision check

[Junio C Hamano]

Taylor Blau <me@ttaylorr.com> writes:

> One of the reasons we never implemented this is because the files it
> moves are all named after the cryptographic hash of their contents
> (either loose objects, or packs which have their hash in the name these
> days).

So we are saying that both loose objects and packfiles would benefit
if we did collision checks here.

> ... So in preparation, let's actually
> implement a byte-for-byte collision check.

But the byte-for-byte collision check is only good for packfiles.

In other words, the definition of "content" that gets hashed to
derive the name can differ among csum-file users, so the way to
check for collision can be written for these different types.

> Note that this may cause some extra computation when the files are in
> fact identical, but this should happen rarely.
>
> Loose objects are exempt from this check, and the collision check may be
> skipped by calling the _flags variant of this function with the
> FOF_SKIP_COLLISION_CHECK bit set. This is done for a couple of reasons:
>
>   - We don't treat the hash of the loose object file's contents as a
>     checksum, since the same loose object can be stored using different
>     bytes on disk (e.g., when adjusting core.compression, using a
>     different version of zlib, etc.).

That's a good explanation why byte-for-byte check is inappropriate.

>   - We already use the path of the loose object as its hash value /
>     object name, so checking for collisions at the content level doesn't
>     add anything.

"We already ... doesn't add anything" -> "The point of collision
check is to find an attempt to store different contents that hashes
down to the same object name, and we continue to use sha1dc to hash
the content name so such a collision would have already been
detected".

>     This is why we do not bother to check the inflated object contents
>     for collisions either, since either (a) the object contents have the
>     fingerprint of a SHA-1 collision, in which case the collision
>     detecting SHA-1 implementation used to hash the contents to give us
>     a path would have already rejected it, or (b) the contents are part
>     of a colliding pair which does not bear the same fingerprints of
>     known collision attacks, in which case we would not have caught it
>     anyway.

I do not understand (b).  If you compare inflated contents, you
would find such a pair that sha1dc missed but the "implement
collision check" patch would have caught as a pair of byte sequences
that hash to the same value.  Isn't it an opportunity to make it safer
to implement such a loose object collision check?

Thanks.

Re: [PATCH v4 3/8] finalize_object_file(): implement collision check

[Taylor Blau]

On Tue, Sep 24, 2024 at 02:32:21PM -0700, Junio C Hamano wrote:
> >   - We already use the path of the loose object as its hash value /
> >     object name, so checking for collisions at the content level doesn't
> >     add anything.
>
> "We already ... doesn't add anything" -> "The point of collision
> check is to find an attempt to store different contents that hashes
> down to the same object name, and we continue to use sha1dc to hash
> the content name so such a collision would have already been
> detected".

Exactly.

> >     This is why we do not bother to check the inflated object contents
> >     for collisions either, since either (a) the object contents have the
> >     fingerprint of a SHA-1 collision, in which case the collision
> >     detecting SHA-1 implementation used to hash the contents to give us
> >     a path would have already rejected it, or (b) the contents are part
> >     of a colliding pair which does not bear the same fingerprints of
> >     known collision attacks, in which case we would not have caught it
> >     anyway.
>
> I do not understand (b).  If you compare inflated contents, you
> would find such a pair that sha1dc missed but the "implement
> collision check" patch would have caught as a pair of byte sequences
> that hash to the same value.  Isn't it an opportunity to make it safer
> to implement such a loose object collision check?

All I was saying with (b) was that if you had some new collision attack
that didn't trigger the existing detection mechanisms in the existing
collision-detecting SHA-1 implementation, then you wouldn't notice the
collision anyway.

You could inflate the contents and compare them, but I don't think it
would much matter at that point since you've already let one of them
enter the repository, and it's not clear which is the "correct" one to
keep since they both hash to the same value.

Thanks,
Taylor

[PATCH v4 4/8] pack-objects: use finalize_object_file() to rename pack/idx/etc

[Taylor Blau]

In most places that write files to the object database (even packfiles
via index-pack or fast-import), we use finalize_object_file(). This
prefers link()/unlink() over rename(), because it means we will prefer
data that is already in the repository to data that we are newly
writing.

We should do the same thing in pack-objects. Even though we don't think
of it as accepting outside data (and thus not being susceptible to
collision attacks), in theory a determined attacker could present just
the right set of objects to cause an incremental repack to generate
a pack with their desired hash.

This has some test and real-world fallout, as seen in the adjustment to
t5303 below. That test script assumes that we can "fix" corruption by
repacking into a good state, including when the pack generated by that
repack operation collides with a (corrupted) pack with the same hash.
This violates our assumption from the previous adjustments to
finalize_object_file() that if we're moving a new file over an existing
one, that since their checksums match, so too must their contents.

This makes "fixing" corruption like this a more explicit operation,
since the test (and users, who may fix real-life corruption using a
similar technique) must first move the broken contents out of the way.

Note also that we now call adjust_shared_perm() twice. We already call
adjust_shared_perm() in stage_tmp_packfiles(), and now call it again in
finalize_object_file(). This is somewhat wasteful, but cleaning up the
existing calls to adjust_shared_perm() is tricky (because sometimes
we're writing to a tmpfile, and sometimes we're writing directly into
the final destination), so let's tolerate some minor waste until we can
more carefully clean up the now-redundant calls.

Co-authored-by: Jeff King <peff@peff.net>
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Taylor Blau <me@ttaylorr.com>
---
 pack-write.c                          | 7 ++++---
 t/t5303-pack-corruption-resilience.sh | 7 ++++++-
 2 files changed, 10 insertions(+), 4 deletions(-)

diff --git a/pack-write.c b/pack-write.c
index 27965672f17..f415604c159 100644
--- a/pack-write.c
+++ b/pack-write.c
@@ -8,6 +8,7 @@
 #include "csum-file.h"
 #include "remote.h"
 #include "chunk-format.h"
+#include "object-file.h"
 #include "pack-mtimes.h"
 #include "pack-objects.h"
 #include "pack-revindex.h"
@@ -528,9 +529,9 @@ static void rename_tmp_packfile(struct strbuf *name_prefix, const char *source,
 	size_t name_prefix_len = name_prefix->len;
 
 	strbuf_addstr(name_prefix, ext);
-	if (rename(source, name_prefix->buf))
-		die_errno("unable to rename temporary file to '%s'",
-			  name_prefix->buf);
+	if (finalize_object_file(source, name_prefix->buf))
+		die("unable to rename temporary file to '%s'",
+		    name_prefix->buf);
 	strbuf_setlen(name_prefix, name_prefix_len);
 }
 
diff --git a/t/t5303-pack-corruption-resilience.sh b/t/t5303-pack-corruption-resilience.sh
index 61469ef4a68..e6a43ec9ae3 100755
--- a/t/t5303-pack-corruption-resilience.sh
+++ b/t/t5303-pack-corruption-resilience.sh
@@ -44,9 +44,14 @@ create_new_pack() {
 }
 
 do_repack() {
+    for f in $pack.*
+    do
+	    mv $f "$(echo $f | sed -e 's/pack-/pack-corrupt-/')" || return 1
+    done &&
     pack=$(printf "$blob_1\n$blob_2\n$blob_3\n" |
           git pack-objects $@ .git/objects/pack/pack) &&
-    pack=".git/objects/pack/pack-${pack}"
+    pack=".git/objects/pack/pack-${pack}" &&
+    rm -f .git/objects/pack/pack-corrupt-*
 }
 
 do_corrupt_object() {
-- 
2.46.2.636.g4b83dd05e9f

Re: [PATCH v4 4/8] pack-objects: use finalize_object_file() to rename pack/idx/etc

[Junio C Hamano]

Taylor Blau <me@ttaylorr.com> writes:

> This has some test and real-world fallout, as seen in the adjustment to
> t5303 below. That test script assumes that we can "fix" corruption by
> repacking into a good state, including when the pack generated by that
> repack operation collides with a (corrupted) pack with the same hash.
> This violates our assumption from the previous adjustments to
> finalize_object_file() that if we're moving a new file over an existing
> one, that since their checksums match, so too must their contents.
>
> This makes "fixing" corruption like this a more explicit operation,
> since the test (and users, who may fix real-life corruption using a
> similar technique) must first move the broken contents out of the way.

Nicely described.

> @@ -528,9 +529,9 @@ static void rename_tmp_packfile(struct strbuf *name_prefix, const char *source,
>  	size_t name_prefix_len = name_prefix->len;
>  
>  	strbuf_addstr(name_prefix, ext);
> -	if (rename(source, name_prefix->buf))
> -		die_errno("unable to rename temporary file to '%s'",
> -			  name_prefix->buf);
> +	if (finalize_object_file(source, name_prefix->buf))
> +		die("unable to rename temporary file to '%s'",
> +		    name_prefix->buf);
>  	strbuf_setlen(name_prefix, name_prefix_len);
>  }

Looking good.

Thanks.

[PATCH v4 5/8] sha1: do not redefine `platform_SHA_CTX` and friends

[Taylor Blau]

Our in-tree SHA-1 wrappers all define platform_SHA_CTX and related
macros to point at the opaque "context" type, init, update, and similar
functions for each specific implementation.

In hash.h, we use these platform_ variables to set up the function
pointers for, e.g., the_hash_algo->init_fn(), etc.

But while these header files have a header-specific macro that prevents
them declaring their structs / functions multiple times, they
unconditionally define the platform variables, making it impossible to
load multiple SHA-1 implementations at once.

As a prerequisite for loading a separate SHA-1 implementation for
non-cryptographic uses, only define the platform_ variables if they have
not already been defined.

Signed-off-by: Taylor Blau <me@ttaylorr.com>
---
 block-sha1/sha1.h | 2 ++
 sha1/openssl.h    | 2 ++
 sha1dc_git.h      | 3 +++
 3 files changed, 7 insertions(+)

diff --git a/block-sha1/sha1.h b/block-sha1/sha1.h
index 9fb0441b988..47bb9166368 100644
--- a/block-sha1/sha1.h
+++ b/block-sha1/sha1.h
@@ -16,7 +16,9 @@ void blk_SHA1_Init(blk_SHA_CTX *ctx);
 void blk_SHA1_Update(blk_SHA_CTX *ctx, const void *dataIn, size_t len);
 void blk_SHA1_Final(unsigned char hashout[20], blk_SHA_CTX *ctx);
 
+#ifndef platform_SHA_CTX
 #define platform_SHA_CTX	blk_SHA_CTX
 #define platform_SHA1_Init	blk_SHA1_Init
 #define platform_SHA1_Update	blk_SHA1_Update
 #define platform_SHA1_Final	blk_SHA1_Final
+#endif
diff --git a/sha1/openssl.h b/sha1/openssl.h
index 006c1f4ba54..1038af47daf 100644
--- a/sha1/openssl.h
+++ b/sha1/openssl.h
@@ -40,10 +40,12 @@ static inline void openssl_SHA1_Clone(struct openssl_SHA1_CTX *dst,
 	EVP_MD_CTX_copy_ex(dst->ectx, src->ectx);
 }
 
+#ifndef platform_SHA_CTX
 #define platform_SHA_CTX openssl_SHA1_CTX
 #define platform_SHA1_Init openssl_SHA1_Init
 #define platform_SHA1_Clone openssl_SHA1_Clone
 #define platform_SHA1_Update openssl_SHA1_Update
 #define platform_SHA1_Final openssl_SHA1_Final
+#endif
 
 #endif /* SHA1_OPENSSL_H */
diff --git a/sha1dc_git.h b/sha1dc_git.h
index 60e3ce84395..f6f880cabea 100644
--- a/sha1dc_git.h
+++ b/sha1dc_git.h
@@ -18,7 +18,10 @@ void git_SHA1DCFinal(unsigned char [20], SHA1_CTX *);
 void git_SHA1DCUpdate(SHA1_CTX *ctx, const void *data, unsigned long len);
 
 #define platform_SHA_IS_SHA1DC /* used by "test-tool sha1-is-sha1dc" */
+
+#ifndef platform_SHA_CTX
 #define platform_SHA_CTX SHA1_CTX
 #define platform_SHA1_Init git_SHA1DCInit
 #define platform_SHA1_Update git_SHA1DCUpdate
 #define platform_SHA1_Final git_SHA1DCFinal
+#endif
-- 
2.46.2.636.g4b83dd05e9f

[PATCH v4 6/8] hash.h: scaffolding for _unsafe hashing variants

[Taylor Blau]

Git's default SHA-1 implementation is collision-detecting, which hardens
us against known SHA-1 attacks against Git objects. This makes Git
object writes safer at the expense of some speed when hashing through
the collision-detecting implementation, which is slower than
non-collision detecting alternatives.

Prepare for loading a separate "unsafe" SHA-1 implementation that can be
used for non-cryptographic purposes, like computing the checksum of
files that use the hashwrite() API.

This commit does not actually introduce any new compile-time knobs to
control which implementation is used as the unsafe SHA-1 variant, but
does add scaffolding so that the "git_hash_algo" structure has five new
function pointers which are "unsafe" variants of the five existing
hashing-related function pointers:

  - git_hash_init_fn unsafe_init_fn
  - git_hash_clone_fn unsafe_clone_fn
  - git_hash_update_fn unsafe_update_fn
  - git_hash_final_fn unsafe_final_fn
  - git_hash_final_oid_fn unsafe_final_oid_fn

The following commit will introduce compile-time knobs to specify which
SHA-1 implementation is used for non-cryptographic uses.

Signed-off-by: Taylor Blau <me@ttaylorr.com>
---
 hash.h        | 42 ++++++++++++++++++++++++++++++++++++++++++
 object-file.c | 42 ++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 84 insertions(+)

diff --git a/hash.h b/hash.h
index 72ffbc862e5..96458b129f9 100644
--- a/hash.h
+++ b/hash.h
@@ -44,14 +44,32 @@
 #define platform_SHA1_Final    	SHA1_Final
 #endif
 
+#ifndef platform_SHA_CTX_unsafe
+#  define platform_SHA_CTX_unsafe      platform_SHA_CTX
+#  define platform_SHA1_Init_unsafe    platform_SHA1_Init
+#  define platform_SHA1_Update_unsafe  platform_SHA1_Update
+#  define platform_SHA1_Final_unsafe   platform_SHA1_Final
+#  ifdef platform_SHA1_Clone
+#    define platform_SHA1_Clone_unsafe platform_SHA1_Clone
+#  endif
+#endif
+
 #define git_SHA_CTX		platform_SHA_CTX
 #define git_SHA1_Init		platform_SHA1_Init
 #define git_SHA1_Update		platform_SHA1_Update
 #define git_SHA1_Final		platform_SHA1_Final
 
+#define git_SHA_CTX_unsafe	platform_SHA_CTX_unsafe
+#define git_SHA1_Init_unsafe	platform_SHA1_Init_unsafe
+#define git_SHA1_Update_unsafe	platform_SHA1_Update_unsafe
+#define git_SHA1_Final_unsafe	platform_SHA1_Final_unsafe
+
 #ifdef platform_SHA1_Clone
 #define git_SHA1_Clone	platform_SHA1_Clone
 #endif
+#ifdef platform_SHA1_Clone_unsafe
+#  define git_SHA1_Clone_unsafe platform_SHA1_Clone_unsafe
+#endif
 
 #ifndef platform_SHA256_CTX
 #define platform_SHA256_CTX	SHA256_CTX
@@ -81,6 +99,13 @@ static inline void git_SHA1_Clone(git_SHA_CTX *dst, const git_SHA_CTX *src)
 	memcpy(dst, src, sizeof(*dst));
 }
 #endif
+#ifndef SHA1_NEEDS_CLONE_HELPER_UNSAFE
+static inline void git_SHA1_Clone_unsafe(git_SHA_CTX_unsafe *dst,
+				       const git_SHA_CTX_unsafe *src)
+{
+	memcpy(dst, src, sizeof(*dst));
+}
+#endif
 
 #ifndef SHA256_NEEDS_CLONE_HELPER
 static inline void git_SHA256_Clone(git_SHA256_CTX *dst, const git_SHA256_CTX *src)
@@ -178,6 +203,8 @@ enum get_oid_result {
 /* A suitably aligned type for stack allocations of hash contexts. */
 union git_hash_ctx {
 	git_SHA_CTX sha1;
+	git_SHA_CTX_unsafe sha1_unsafe;
+
 	git_SHA256_CTX sha256;
 };
 typedef union git_hash_ctx git_hash_ctx;
@@ -222,6 +249,21 @@ struct git_hash_algo {
 	/* The hash finalization function for object IDs. */
 	git_hash_final_oid_fn final_oid_fn;
 
+	/* The non-cryptographic hash initialization function. */
+	git_hash_init_fn unsafe_init_fn;
+
+	/* The non-cryptographic hash context cloning function. */
+	git_hash_clone_fn unsafe_clone_fn;
+
+	/* The non-cryptographic hash update function. */
+	git_hash_update_fn unsafe_update_fn;
+
+	/* The non-cryptographic hash finalization function. */
+	git_hash_final_fn unsafe_final_fn;
+
+	/* The non-cryptographic hash finalization function. */
+	git_hash_final_oid_fn unsafe_final_oid_fn;
+
 	/* The OID of the empty tree. */
 	const struct object_id *empty_tree;
 
diff --git a/object-file.c b/object-file.c
index b9a3a1f62db..196c9e2df8b 100644
--- a/object-file.c
+++ b/object-file.c
@@ -115,6 +115,33 @@ static void git_hash_sha1_final_oid(struct object_id *oid, git_hash_ctx *ctx)
 	oid->algo = GIT_HASH_SHA1;
 }
 
+static void git_hash_sha1_init_unsafe(git_hash_ctx *ctx)
+{
+	git_SHA1_Init_unsafe(&ctx->sha1_unsafe);
+}
+
+static void git_hash_sha1_clone_unsafe(git_hash_ctx *dst, const git_hash_ctx *src)
+{
+	git_SHA1_Clone_unsafe(&dst->sha1_unsafe, &src->sha1_unsafe);
+}
+
+static void git_hash_sha1_update_unsafe(git_hash_ctx *ctx, const void *data,
+				      size_t len)
+{
+	git_SHA1_Update_unsafe(&ctx->sha1_unsafe, data, len);
+}
+
+static void git_hash_sha1_final_unsafe(unsigned char *hash, git_hash_ctx *ctx)
+{
+	git_SHA1_Final_unsafe(hash, &ctx->sha1_unsafe);
+}
+
+static void git_hash_sha1_final_oid_unsafe(struct object_id *oid, git_hash_ctx *ctx)
+{
+	git_SHA1_Final_unsafe(oid->hash, &ctx->sha1_unsafe);
+	memset(oid->hash + GIT_SHA1_RAWSZ, 0, GIT_MAX_RAWSZ - GIT_SHA1_RAWSZ);
+	oid->algo = GIT_HASH_SHA1;
+}
 
 static void git_hash_sha256_init(git_hash_ctx *ctx)
 {
@@ -189,6 +216,11 @@ const struct git_hash_algo hash_algos[GIT_HASH_NALGOS] = {
 		.update_fn = git_hash_unknown_update,
 		.final_fn = git_hash_unknown_final,
 		.final_oid_fn = git_hash_unknown_final_oid,
+		.unsafe_init_fn = git_hash_unknown_init,
+		.unsafe_clone_fn = git_hash_unknown_clone,
+		.unsafe_update_fn = git_hash_unknown_update,
+		.unsafe_final_fn = git_hash_unknown_final,
+		.unsafe_final_oid_fn = git_hash_unknown_final_oid,
 		.empty_tree = NULL,
 		.empty_blob = NULL,
 		.null_oid = NULL,
@@ -204,6 +236,11 @@ const struct git_hash_algo hash_algos[GIT_HASH_NALGOS] = {
 		.update_fn = git_hash_sha1_update,
 		.final_fn = git_hash_sha1_final,
 		.final_oid_fn = git_hash_sha1_final_oid,
+		.unsafe_init_fn = git_hash_sha1_init_unsafe,
+		.unsafe_clone_fn = git_hash_sha1_clone_unsafe,
+		.unsafe_update_fn = git_hash_sha1_update_unsafe,
+		.unsafe_final_fn = git_hash_sha1_final_unsafe,
+		.unsafe_final_oid_fn = git_hash_sha1_final_oid_unsafe,
 		.empty_tree = &empty_tree_oid,
 		.empty_blob = &empty_blob_oid,
 		.null_oid = &null_oid_sha1,
@@ -219,6 +256,11 @@ const struct git_hash_algo hash_algos[GIT_HASH_NALGOS] = {
 		.update_fn = git_hash_sha256_update,
 		.final_fn = git_hash_sha256_final,
 		.final_oid_fn = git_hash_sha256_final_oid,
+		.unsafe_init_fn = git_hash_sha256_init,
+		.unsafe_clone_fn = git_hash_sha256_clone,
+		.unsafe_update_fn = git_hash_sha256_update,
+		.unsafe_final_fn = git_hash_sha256_final,
+		.unsafe_final_oid_fn = git_hash_sha256_final_oid,
 		.empty_tree = &empty_tree_oid_sha256,
 		.empty_blob = &empty_blob_oid_sha256,
 		.null_oid = &null_oid_sha256,
-- 
2.46.2.636.g4b83dd05e9f

[PATCH v4 7/8] Makefile: allow specifying a SHA-1 for non-cryptographic uses

[Taylor Blau]

Introduce _UNSAFE variants of the OPENSSL_SHA1, BLK_SHA1, and
APPLE_COMMON_CRYPTO_SHA1 compile-time knobs which indicate which SHA-1
implementation is to be used for non-cryptographic uses.

There are a couple of small implementation notes worth mentioning:

  - There is no way to select the collision detecting SHA-1 as the
    "fast" fallback, since the fast fallback is only for
    non-cryptographic uses, and is meant to be faster than our
    collision-detecting implementation.

  - There are no similar knobs for SHA-256, since no collision attacks
    are presently known and thus no collision-detecting implementations
    actually exist.

Signed-off-by: Taylor Blau <me@ttaylorr.com>
---
 Makefile | 25 +++++++++++++++++++++++++
 hash.h   | 30 ++++++++++++++++++++++++++++++
 2 files changed, 55 insertions(+)

diff --git a/Makefile b/Makefile
index 9cf2be070fa..fb84a87592d 100644
--- a/Makefile
+++ b/Makefile
@@ -521,6 +521,10 @@ include shared.mak
 # Define APPLE_COMMON_CRYPTO_SHA1 to use Apple's CommonCrypto for
 # SHA-1.
 #
+# Define the same Makefile knobs as above, but suffixed with _UNSAFE to
+# use the corresponding implementations for unsafe SHA-1 hashing for
+# non-cryptographic purposes.
+#
 # If don't enable any of the *_SHA1 settings in this section, Git will
 # default to its built-in sha1collisiondetection library, which is a
 # collision-detecting sha1 This is slower, but may detect attempted
@@ -1996,6 +2000,27 @@ endif
 endif
 endif
 
+ifdef OPENSSL_SHA1_UNSAFE
+ifndef OPENSSL_SHA1
+	EXTLIBS += $(LIB_4_CRYPTO)
+	BASIC_CFLAGS += -DSHA1_OPENSSL_UNSAFE
+endif
+else
+ifdef BLK_SHA1_UNSAFE
+ifndef BLK_SHA1
+	LIB_OBJS += block-sha1/sha1.o
+	BASIC_CFLAGS += -DSHA1_BLK_UNSAFE
+endif
+else
+ifdef APPLE_COMMON_CRYPTO_SHA1_UNSAFE
+ifndef APPLE_COMMON_CRYPTO_SHA1
+	COMPAT_CFLAGS += -DCOMMON_DIGEST_FOR_OPENSSL
+	BASIC_CFLAGS += -DSHA1_APPLE_UNSAFE
+endif
+endif
+endif
+endif
+
 ifdef OPENSSL_SHA256
 	EXTLIBS += $(LIB_4_CRYPTO)
 	BASIC_CFLAGS += -DSHA256_OPENSSL
diff --git a/hash.h b/hash.h
index 96458b129f9..f97f858307b 100644
--- a/hash.h
+++ b/hash.h
@@ -15,6 +15,36 @@
 #include "block-sha1/sha1.h"
 #endif
 
+#if defined(SHA1_APPLE_UNSAFE)
+#  include <CommonCrypto/CommonDigest.h>
+#  define platform_SHA_CTX_unsafe CC_SHA1_CTX
+#  define platform_SHA1_Init_unsafe CC_SHA1_Init
+#  define platform_SHA1_Update_unsafe CC_SHA1_Update
+#  define platform_SHA1_Final_unsafe CC_SHA1_Final
+#elif defined(SHA1_OPENSSL_UNSAFE)
+#  include <openssl/sha.h>
+#  if defined(OPENSSL_API_LEVEL) && OPENSSL_API_LEVEL >= 3
+#    define SHA1_NEEDS_CLONE_HELPER_UNSAFE
+#    include "sha1/openssl.h"
+#    define platform_SHA_CTX_unsafe openssl_SHA1_CTX
+#    define platform_SHA1_Init_unsafe openssl_SHA1_Init
+#    define platform_SHA1_Clone_unsafe openssl_SHA1_Clone
+#    define platform_SHA1_Update_unsafe openssl_SHA1_Update
+#    define platform_SHA1_Final_unsafe openssl_SHA1_Final
+#  else
+#    define platform_SHA_CTX_unsafe SHA_CTX
+#    define platform_SHA1_Init_unsafe SHA1_Init
+#    define platform_SHA1_Update_unsafe SHA1_Update
+#    define platform_SHA1_Final_unsafe SHA1_Final
+#  endif
+#elif defined(SHA1_BLK_UNSAFE)
+#  include "block-sha1/sha1.h"
+#  define platform_SHA_CTX_unsafe blk_SHA_CTX
+#  define platform_SHA1_Init_unsafe blk_SHA1_Init
+#  define platform_SHA1_Update_unsafe blk_SHA1_Update
+#  define platform_SHA1_Final_unsafe blk_SHA1_Final
+#endif
+
 #if defined(SHA256_NETTLE)
 #include "sha256/nettle.h"
 #elif defined(SHA256_GCRYPT)
-- 
2.46.2.636.g4b83dd05e9f

[PATCH v4 8/8] csum-file.c: use unsafe SHA-1 implementation when available

[Taylor Blau]

Update hashwrite() and friends to use the unsafe_-variants of hashing
functions, calling for e.g., "the_hash_algo->unsafe_update_fn()" instead
of "the_hash_algo->update_fn()".

These callers only use the_hash_algo to produce a checksum, which we
depend on for data integrity, but not for cryptographic purposes, so
these callers are safe to use the unsafe (and potentially non-collision
detecting) SHA-1 implementation.

To time this, I took a freshly packed copy of linux.git, and ran the
following with and without the OPENSSL_SHA1_UNSAFE=1 build-knob. Both
versions were compiled with -O3:

    $ git for-each-ref --format='%(objectname)' refs/heads refs/tags >in
    $ valgrind --tool=callgrind ~/src/git/git-pack-objects \
        --revs --stdout --all-progress --use-bitmap-index <in >/dev/null

Without OPENSSL_SHA1_UNSAFE=1 (that is, using the collision-detecting
SHA-1 implementation for both cryptographic and non-cryptographic
purposes), we spend a significant amount of our instruction count in
hashwrite():

    $ callgrind_annotate --inclusive=yes | grep hashwrite | head -n1
    159,998,868,413 (79.42%)  /home/ttaylorr/src/git/csum-file.c:hashwrite [/home/ttaylorr/src/git/git-pack-objects]

, and the resulting "clone" takes 19.219 seconds of wall clock time,
18.94 seconds of user time and 0.28 seconds of system time.

Compiling with OPENSSL_SHA1_UNSAFE=1, we spend ~60% fewer instructions
in hashwrite():

    $ callgrind_annotate --inclusive=yes | grep hashwrite | head -n1
     59,164,001,176 (58.79%)  /home/ttaylorr/src/git/csum-file.c:hashwrite [/home/ttaylorr/src/git/git-pack-objects]

, and generate the resulting "clone" much unsafeer, in only 11.597 seconds
of wall time, 11.37 seconds of user time, and 0.23 seconds of system
time, for a ~40% speed-up.

Signed-off-by: Taylor Blau <me@ttaylorr.com>
---
 csum-file.c | 18 +++++++++---------
 1 file changed, 9 insertions(+), 9 deletions(-)

diff --git a/csum-file.c b/csum-file.c
index bf82ad8f9f5..c203ebf11b3 100644
--- a/csum-file.c
+++ b/csum-file.c
@@ -50,7 +50,7 @@ void hashflush(struct hashfile *f)
 
 	if (offset) {
 		if (!f->skip_hash)
-			the_hash_algo->update_fn(&f->ctx, f->buffer, offset);
+			the_hash_algo->unsafe_update_fn(&f->ctx, f->buffer, offset);
 		flush(f, f->buffer, offset);
 		f->offset = 0;
 	}
@@ -73,7 +73,7 @@ int finalize_hashfile(struct hashfile *f, unsigned char *result,
 	if (f->skip_hash)
 		hashclr(f->buffer, the_repository->hash_algo);
 	else
-		the_hash_algo->final_fn(f->buffer, &f->ctx);
+		the_hash_algo->unsafe_final_fn(f->buffer, &f->ctx);
 
 	if (result)
 		hashcpy(result, f->buffer, the_repository->hash_algo);
@@ -128,7 +128,7 @@ void hashwrite(struct hashfile *f, const void *buf, unsigned int count)
 			 * f->offset is necessarily zero.
 			 */
 			if (!f->skip_hash)
-				the_hash_algo->update_fn(&f->ctx, buf, nr);
+				the_hash_algo->unsafe_update_fn(&f->ctx, buf, nr);
 			flush(f, buf, nr);
 		} else {
 			/*
@@ -174,7 +174,7 @@ static struct hashfile *hashfd_internal(int fd, const char *name,
 	f->name = name;
 	f->do_crc = 0;
 	f->skip_hash = 0;
-	the_hash_algo->init_fn(&f->ctx);
+	the_hash_algo->unsafe_init_fn(&f->ctx);
 
 	f->buffer_len = buffer_len;
 	f->buffer = xmalloc(buffer_len);
@@ -208,7 +208,7 @@ void hashfile_checkpoint(struct hashfile *f, struct hashfile_checkpoint *checkpo
 {
 	hashflush(f);
 	checkpoint->offset = f->total;
-	the_hash_algo->clone_fn(&checkpoint->ctx, &f->ctx);
+	the_hash_algo->unsafe_clone_fn(&checkpoint->ctx, &f->ctx);
 }
 
 int hashfile_truncate(struct hashfile *f, struct hashfile_checkpoint *checkpoint)
@@ -219,7 +219,7 @@ int hashfile_truncate(struct hashfile *f, struct hashfile_checkpoint *checkpoint
 	    lseek(f->fd, offset, SEEK_SET) != offset)
 		return -1;
 	f->total = offset;
-	the_hash_algo->clone_fn(&f->ctx, &checkpoint->ctx);
+	the_hash_algo->unsafe_clone_fn(&f->ctx, &checkpoint->ctx);
 	f->offset = 0; /* hashflush() was called in checkpoint */
 	return 0;
 }
@@ -245,9 +245,9 @@ int hashfile_checksum_valid(const unsigned char *data, size_t total_len)
 	if (total_len < the_hash_algo->rawsz)
 		return 0; /* say "too short"? */
 
-	the_hash_algo->init_fn(&ctx);
-	the_hash_algo->update_fn(&ctx, data, data_len);
-	the_hash_algo->final_fn(got, &ctx);
+	the_hash_algo->unsafe_init_fn(&ctx);
+	the_hash_algo->unsafe_update_fn(&ctx, data, data_len);
+	the_hash_algo->unsafe_final_fn(got, &ctx);
 
 	return hasheq(got, data + data_len, the_repository->hash_algo);
 }
-- 
2.46.2.636.g4b83dd05e9f

Re: [PATCH v4 0/8] hash.h: support choosing a separate SHA-1 for non-cryptographic uses

[Jeff King]

On Tue, Sep 24, 2024 at 01:32:01PM -0400, Taylor Blau wrote:

> This version is another fairly substantial reroll, with the main
> differences being renaming the "fast" SHA-1 options to "unsafe", as well
> as not running the collision checks via finalize_object_file() when
> handling loose objects (see the relevant patches for details on why).

I left some rambling comments on the collision check changes.  Nothing
earth-shattering, but you may or may not want to tweak based on what I
said.

I'm happy with the s/fast/unsafe/ rename.

The rest of it was more or less the same, and looks good to me. I do
think with the extra collision-check it would be OK to teach index-pack
to use the fast sha1, too. That would speed up receiving objects by the
same absolute numbers (but less as a relative portion, since delta
resolution is usually much more expensive). And it would also make
verify-pack a bit faster, too (IIRC fsck for some reason does not use
verify-pack, so its code is a potential candidate, too, if it's not
already using csum-file).

Those don't strictly need to come now, but it seems like they might be
worthwhile while we're in the area (OTOH, using the fast hash only when
sending is a belt-and-suspenders with the collision check).

> Note also there is an important bug fix in finalize_object_file() to
> unlink() the temporary file when we do run the collision check, but no
> collisions were found. This bug was causing a pile-up of tmp_obj_XXXXXX
> files in GitHub's infrastructure.

Oops. :) I wondered if we could have a test here, but I don't think this
can be easily triggered in the tests. The loose object collision happens
only via TOCTOU race. I also tried instrumenting the code like below,
and it triggers zero times in the test suite.

diff --git a/object-file.c b/object-file.c
index b9a3a1f62d..d9172df8d9 100644
--- a/object-file.c
+++ b/object-file.c
@@ -40,6 +40,7 @@
 #include "fsck.h"
 #include "loose.h"
 #include "object-file-convert.h"
+#include "trace.h"
 
 /* The maximum size for an object header. */
 #define MAX_HEADER_LEN 32
@@ -1994,6 +1995,7 @@ int finalize_object_file(const char *tmpfile, const char *filename)
 int finalize_object_file_flags(const char *tmpfile, const char *filename,
 			       enum finalize_object_file_flags flags)
 {
+	static struct trace_key t = TRACE_KEY_INIT(COLLISION);
 	struct stat st;
 	int ret = 0;
 
@@ -2031,6 +2033,8 @@ int finalize_object_file_flags(const char *tmpfile, const char *filename,
 			errno = saved_errno;
 			return error_errno(_("unable to write file %s"), filename);
 		}
+		if (flags & FOF_SKIP_COLLISION_CHECK)
+			trace_printf_key(&t, "skipping check of %s and %s", tmpfile, filename);
 		if (!(flags & FOF_SKIP_COLLISION_CHECK) &&
 		    check_collision(tmpfile, filename))
 				return -1;

Re: [PATCH v4 0/8] hash.h: support choosing a separate SHA-1 for non-cryptographic uses

[Elijah Newren]

On Tue, Sep 24, 2024 at 10:32 AM Taylor Blau <me@ttaylorr.com> wrote:
>
> Here is a reroll of mine and Peff's series to add a build-time knob to
> allow selecting an alternative SHA-1 implementation for
> non-cryptographic hashing within Git, starting with the `hashwrite()`
> family of functions.
>
> This version is another fairly substantial reroll, with the main
> differences being renaming the "fast" SHA-1 options to "unsafe", as well
> as not running the collision checks via finalize_object_file() when
> handling loose objects (see the relevant patches for details on why).
>
> Note also there is an important bug fix in finalize_object_file() to
> unlink() the temporary file when we do run the collision check, but no
> collisions were found. This bug was causing a pile-up of tmp_obj_XXXXXX
> files in GitHub's infrastructure.

Yup...

>
> Thanks in advance for your review!
>
> Taylor Blau (8):
>   finalize_object_file(): check for name collision before renaming
>   finalize_object_file(): refactor unlink_or_warn() placement
>   finalize_object_file(): implement collision check
>   pack-objects: use finalize_object_file() to rename pack/idx/etc
>   sha1: do not redefine `platform_SHA_CTX` and friends
>   hash.h: scaffolding for _unsafe hashing variants
>   Makefile: allow specifying a SHA-1 for non-cryptographic uses
>   csum-file.c: use unsafe SHA-1 implementation when available
>
>  Makefile                              |  25 ++++++
>  block-sha1/sha1.h                     |   2 +
>  csum-file.c                           |  18 ++--
>  hash.h                                |  72 +++++++++++++++
>  object-file.c                         | 124 ++++++++++++++++++++++++--
>  object-file.h                         |   6 ++
>  pack-write.c                          |   7 +-
>  sha1/openssl.h                        |   2 +
>  sha1dc_git.h                          |   3 +
>  t/t5303-pack-corruption-resilience.sh |   7 +-
>  tmp-objdir.c                          |  26 ++++--
>  11 files changed, 266 insertions(+), 26 deletions(-)
>
> Range-diff against v3:
>  1:  738b1eb17b4 =  1:  6f1ee91fff3 finalize_object_file(): check for name collision before renaming
>  2:  e1c2c39711f =  2:  133047ca8c9 finalize_object_file(): refactor unlink_or_warn() placement
>  3:  0feee5d1d4f !  3:  ed9eeef8513 finalize_object_file(): implement collision check
>     @@ Commit message
>
>          The new check will cause the write of new differing content to be a
>          failure, rather than a silent noop, and we'll retain the temporary file
>     -    on disk.
>     +    on disk. If there's no collision present, we'll clean up the temporary
>     +    file as usual after either rename()-ing or link()-ing it into place.
>
>          Note that this may cause some extra computation when the files are in
>     -    fact identical, but this should happen rarely. For example, when writing
>     -    a loose object, we compute the object id first, then check manually for
>     -    an existing object (so that we can freshen its timestamp) before
>     -    actually trying to write and link the new data.
>     +    fact identical, but this should happen rarely.
>     +
>     +    Loose objects are exempt from this check, and the collision check may be
>     +    skipped by calling the _flags variant of this function with the
>     +    FOF_SKIP_COLLISION_CHECK bit set. This is done for a couple of reasons:
>     +
>     +      - We don't treat the hash of the loose object file's contents as a
>     +        checksum, since the same loose object can be stored using different
>     +        bytes on disk (e.g., when adjusting core.compression, using a
>     +        different version of zlib, etc.).
>     +
>     +        This is fundamentally different from cases where
>     +        finalize_object_file() is operating over a file which uses the hash
>     +        value as a checksum of the contents. In other words, a pair of
>     +        identical loose objects can be stored using different bytes on disk,
>     +        and that should not be treated as a collision.
>     +
>     +      - We already use the path of the loose object as its hash value /
>     +        object name, so checking for collisions at the content level doesn't
>     +        add anything.
>     +
>     +        This is why we do not bother to check the inflated object contents
>     +        for collisions either, since either (a) the object contents have the
>     +        fingerprint of a SHA-1 collision, in which case the collision
>     +        detecting SHA-1 implementation used to hash the contents to give us
>     +        a path would have already rejected it, or (b) the contents are part
>     +        of a colliding pair which does not bear the same fingerprints of
>     +        known collision attacks, in which case we would not have caught it
>     +        anyway.
>     +
>     +        So skipping the collision check here does not change for better or
>     +        worse the hardness of loose object writes.
>     +
>     +    As a small note related to the latter bullet point above, we must teach
>     +    the tmp-objdir routines to similarly skip the content-level collision
>     +    checks when calling migrate_one() on a loose object file, which we do by
>     +    setting the FOF_SKIP_COLLISION_CHECK bit when we are inside of a loose
>     +    object shard.
>
>          Co-authored-by: Jeff King <peff@peff.net>
>          Signed-off-by: Jeff King <peff@peff.net>

I don't even get a Helped-by for finding the cause of the pile-up of
temporary object files and the suggested tweak to this patch?  ;-)

>     @@ object-file.c: static void write_object_file_prepare_literally(const struct git_
>       /*
>        * Move the just written object into its final resting place.
>        */
>     + int finalize_object_file(const char *tmpfile, const char *filename)
>     ++{
>     ++  return finalize_object_file_flags(tmpfile, filename, 0);
>     ++}
>     ++
>     ++int finalize_object_file_flags(const char *tmpfile, const char *filename,
>     ++                         enum finalize_object_file_flags flags)
>     + {
>     +   struct stat st;
>     +   int ret = 0;
>      @@ object-file.c: int finalize_object_file(const char *tmpfile, const char *filename)
>                         errno = saved_errno;
>                         return error_errno(_("unable to write file %s"), filename);
>                 }
>      -          /* FIXME!!! Collision check here ? */
>     --          unlink_or_warn(tmpfile);
>     -+          if (check_collision(tmpfile, filename))
>     -+                  return -1;
>     ++          if (!(flags & FOF_SKIP_COLLISION_CHECK) &&
>     ++              check_collision(tmpfile, filename))
>     ++                          return -1;
>     +           unlink_or_warn(tmpfile);

My suggested tweak had been simpler and didn't have a way to skip the
collision check, but I like the improvement you've made here; the
explanation you give for allowing it to be skipped makes sense to me.

>         }
>
>     - out:
>     +@@ object-file.c: static int write_loose_object(const struct object_id *oid, char *hdr,
>     +                   warning_errno(_("failed utime() on %s"), tmp_file.buf);
>     +   }
>     +
>     +-  return finalize_object_file(tmp_file.buf, filename.buf);
>     ++  return finalize_object_file_flags(tmp_file.buf, filename.buf,
>     ++                                    FOF_SKIP_COLLISION_CHECK);
>     + }
>     +
>     + static int freshen_loose_object(const struct object_id *oid)
>     +@@ object-file.c: int stream_loose_object(struct input_stream *in_stream, size_t len,
>     +           strbuf_release(&dir);
>     +   }
>     +
>     +-  err = finalize_object_file(tmp_file.buf, filename.buf);
>     ++  err = finalize_object_file_flags(tmp_file.buf, filename.buf,
>     ++                                   FOF_SKIP_COLLISION_CHECK);
>     +   if (!err && compat)
>     +           err = repo_add_loose_object_map(the_repository, oid, &compat_oid);
>     + cleanup:
>     +
>     + ## object-file.h ##
>     +@@ object-file.h: int check_object_signature(struct repository *r, const struct object_id *oid,
>     +  */
>     + int stream_object_signature(struct repository *r, const struct object_id *oid);
>     +
>     ++enum finalize_object_file_flags {
>     ++  FOF_SKIP_COLLISION_CHECK = 1,
>     ++};
>     ++
>     + int finalize_object_file(const char *tmpfile, const char *filename);
>     ++int finalize_object_file_flags(const char *tmpfile, const char *filename,
>     ++                         enum finalize_object_file_flags flags);
>     +
>     + /* Helper to check and "touch" a file */
>     + int check_and_freshen_file(const char *fn, int freshen);
>     +
>     + ## tmp-objdir.c ##
>     +@@ tmp-objdir.c: static int read_dir_paths(struct string_list *out, const char *path)
>     +   return 0;
>     + }
>     +
>     +-static int migrate_paths(struct strbuf *src, struct strbuf *dst);
>     ++static int migrate_paths(struct strbuf *src, struct strbuf *dst,
>     ++                   enum finalize_object_file_flags flags);
>     +
>     +-static int migrate_one(struct strbuf *src, struct strbuf *dst)
>     ++static int migrate_one(struct strbuf *src, struct strbuf *dst,
>     ++                 enum finalize_object_file_flags flags)
>     + {
>     +   struct stat st;
>     +
>     +@@ tmp-objdir.c: static int migrate_one(struct strbuf *src, struct strbuf *dst)
>     +                           return -1;
>     +           } else if (errno != EEXIST)
>     +                   return -1;
>     +-          return migrate_paths(src, dst);
>     ++          return migrate_paths(src, dst, flags);
>     +   }
>     +-  return finalize_object_file(src->buf, dst->buf);
>     ++  return finalize_object_file_flags(src->buf, dst->buf, flags);
>     + }
>     +
>     +-static int migrate_paths(struct strbuf *src, struct strbuf *dst)
>     ++static int is_loose_object_shard(const char *name)
>     ++{
>     ++  return strlen(name) == 2 && isxdigit(name[0]) && isxdigit(name[1]);
>     ++}
>     ++
>     ++static int migrate_paths(struct strbuf *src, struct strbuf *dst,
>     ++                   enum finalize_object_file_flags flags)
>     + {
>     +   size_t src_len = src->len, dst_len = dst->len;
>     +   struct string_list paths = STRING_LIST_INIT_DUP;
>     +@@ tmp-objdir.c: static int migrate_paths(struct strbuf *src, struct strbuf *dst)
>     +
>     +   for (i = 0; i < paths.nr; i++) {
>     +           const char *name = paths.items[i].string;
>     ++          enum finalize_object_file_flags flags_copy = flags;
>     +
>     +           strbuf_addf(src, "/%s", name);
>     +           strbuf_addf(dst, "/%s", name);
>     +
>     +-          ret |= migrate_one(src, dst);
>     ++          if (is_loose_object_shard(name))
>     ++                  flags_copy |= FOF_SKIP_COLLISION_CHECK;
>     ++
>     ++          ret |= migrate_one(src, dst, flags_copy);
>     +
>     +           strbuf_setlen(src, src_len);
>     +           strbuf_setlen(dst, dst_len);
>     +@@ tmp-objdir.c: int tmp_objdir_migrate(struct tmp_objdir *t)
>     +   strbuf_addbuf(&src, &t->path);
>     +   strbuf_addstr(&dst, repo_get_object_directory(the_repository));
>     +
>     +-  ret = migrate_paths(&src, &dst);
>     ++  ret = migrate_paths(&src, &dst, 0);

The rest of the changes here are just to plumb the new collision check
skipping flag through; makes sense.

>     +
>     +   strbuf_release(&src);
>     +   strbuf_release(&dst);
>  4:  620dde48a9d =  4:  3cc7f7b1f67 pack-objects: use finalize_object_file() to rename pack/idx/etc
>  5:  bfe992765cd <  -:  ----------- i5500-git-daemon.sh: use compile-able version of Git without OpenSSL
>  6:  22863d9f6df =  5:  8f8ac0f5b0e sha1: do not redefine `platform_SHA_CTX` and friends
>  7:  119c318d812 !  6:  d300e9c6887 hash.h: scaffolding for _fast hashing variants
>     @@ Metadata
>      Author: Taylor Blau <me@ttaylorr.com>
>
>       ## Commit message ##
>     -    hash.h: scaffolding for _fast hashing variants
>     +    hash.h: scaffolding for _unsafe hashing variants
>
>          Git's default SHA-1 implementation is collision-detecting, which hardens
>          us against known SHA-1 attacks against Git objects. This makes Git
>     @@ Commit message
>          the collision-detecting implementation, which is slower than
>          non-collision detecting alternatives.
>
>     -    Prepare for loading a separate "fast" SHA-1 implementation that can be
>     +    Prepare for loading a separate "unsafe" SHA-1 implementation that can be
>          used for non-cryptographic purposes, like computing the checksum of
>          files that use the hashwrite() API.
>
>          This commit does not actually introduce any new compile-time knobs to
>     -    control which implementation is used as the fast SHA-1 variant, but does
>     -    add scaffolding so that the "git_hash_algo" structure has five new
>     -    function pointers which are "fast" variants of the five existing
>     +    control which implementation is used as the unsafe SHA-1 variant, but
>     +    does add scaffolding so that the "git_hash_algo" structure has five new
>     +    function pointers which are "unsafe" variants of the five existing
>          hashing-related function pointers:
>
>     -      - git_hash_init_fn fast_init_fn
>     -      - git_hash_clone_fn fast_clone_fn
>     -      - git_hash_update_fn fast_update_fn
>     -      - git_hash_final_fn fast_final_fn
>     -      - git_hash_final_oid_fn fast_final_oid_fn
>     +      - git_hash_init_fn unsafe_init_fn
>     +      - git_hash_clone_fn unsafe_clone_fn
>     +      - git_hash_update_fn unsafe_update_fn
>     +      - git_hash_final_fn unsafe_final_fn
>     +      - git_hash_final_oid_fn unsafe_final_oid_fn
>
>          The following commit will introduce compile-time knobs to specify which
>          SHA-1 implementation is used for non-cryptographic uses.
>     @@ hash.h
>       #define platform_SHA1_Final       SHA1_Final
>       #endif
>
>     -+#ifndef platform_SHA_CTX_fast
>     -+#  define platform_SHA_CTX_fast     platform_SHA_CTX
>     -+#  define platform_SHA1_Init_fast   platform_SHA1_Init
>     -+#  define platform_SHA1_Update_fast platform_SHA1_Update
>     -+#  define platform_SHA1_Final_fast  platform_SHA1_Final
>     ++#ifndef platform_SHA_CTX_unsafe
>     ++#  define platform_SHA_CTX_unsafe      platform_SHA_CTX
>     ++#  define platform_SHA1_Init_unsafe    platform_SHA1_Init
>     ++#  define platform_SHA1_Update_unsafe  platform_SHA1_Update
>     ++#  define platform_SHA1_Final_unsafe   platform_SHA1_Final
>      +#  ifdef platform_SHA1_Clone
>     -+#    define platform_SHA1_Clone_fast platform_SHA1_Clone
>     ++#    define platform_SHA1_Clone_unsafe platform_SHA1_Clone
>      +#  endif
>      +#endif
>      +
>     @@ hash.h
>       #define git_SHA1_Update           platform_SHA1_Update
>       #define git_SHA1_Final            platform_SHA1_Final
>
>     -+#define git_SHA_CTX_fast  platform_SHA_CTX_fast
>     -+#define git_SHA1_Init_fast        platform_SHA1_Init_fast
>     -+#define git_SHA1_Update_fast      platform_SHA1_Update_fast
>     -+#define git_SHA1_Final_fast       platform_SHA1_Final_fast
>     ++#define git_SHA_CTX_unsafe        platform_SHA_CTX_unsafe
>     ++#define git_SHA1_Init_unsafe      platform_SHA1_Init_unsafe
>     ++#define git_SHA1_Update_unsafe    platform_SHA1_Update_unsafe
>     ++#define git_SHA1_Final_unsafe     platform_SHA1_Final_unsafe
>      +
>       #ifdef platform_SHA1_Clone
>       #define git_SHA1_Clone    platform_SHA1_Clone
>       #endif
>     -+#ifdef platform_SHA1_Clone_fast
>     -+#  define git_SHA1_Clone_fast platform_SHA1_Clone_fast
>     ++#ifdef platform_SHA1_Clone_unsafe
>     ++#  define git_SHA1_Clone_unsafe platform_SHA1_Clone_unsafe
>      +#endif
>
>       #ifndef platform_SHA256_CTX
>     @@ hash.h: static inline void git_SHA1_Clone(git_SHA_CTX *dst, const git_SHA_CTX *s
>         memcpy(dst, src, sizeof(*dst));
>       }
>       #endif
>     -+#ifndef SHA1_NEEDS_CLONE_HELPER_FAST
>     -+static inline void git_SHA1_Clone_fast(git_SHA_CTX_fast *dst,
>     -+                                 const git_SHA_CTX_fast *src)
>     ++#ifndef SHA1_NEEDS_CLONE_HELPER_UNSAFE
>     ++static inline void git_SHA1_Clone_unsafe(git_SHA_CTX_unsafe *dst,
>     ++                                 const git_SHA_CTX_unsafe *src)
>      +{
>      +  memcpy(dst, src, sizeof(*dst));
>      +}
>     @@ hash.h: enum get_oid_result {
>       /* A suitably aligned type for stack allocations of hash contexts. */
>       union git_hash_ctx {
>         git_SHA_CTX sha1;
>     -+  git_SHA_CTX_fast sha1_fast;
>     ++  git_SHA_CTX_unsafe sha1_unsafe;
>      +
>         git_SHA256_CTX sha256;
>       };
>     @@ hash.h: struct git_hash_algo {
>         /* The hash finalization function for object IDs. */
>         git_hash_final_oid_fn final_oid_fn;
>
>     -+  /* The fast / non-cryptographic hash initialization function. */
>     -+  git_hash_init_fn fast_init_fn;
>     ++  /* The non-cryptographic hash initialization function. */
>     ++  git_hash_init_fn unsafe_init_fn;
>      +
>     -+  /* The fast / non-cryptographic hash context cloning function. */
>     -+  git_hash_clone_fn fast_clone_fn;
>     ++  /* The non-cryptographic hash context cloning function. */
>     ++  git_hash_clone_fn unsafe_clone_fn;
>      +
>     -+  /* The fast / non-cryptographic hash update function. */
>     -+  git_hash_update_fn fast_update_fn;
>     ++  /* The non-cryptographic hash update function. */
>     ++  git_hash_update_fn unsafe_update_fn;
>      +
>     -+  /* The fast / non-cryptographic hash finalization function. */
>     -+  git_hash_final_fn fast_final_fn;
>     ++  /* The non-cryptographic hash finalization function. */
>     ++  git_hash_final_fn unsafe_final_fn;
>      +
>     -+  /* The fast / non-cryptographic hash finalization function. */
>     -+  git_hash_final_oid_fn fast_final_oid_fn;
>     ++  /* The non-cryptographic hash finalization function. */
>     ++  git_hash_final_oid_fn unsafe_final_oid_fn;
>      +
>         /* The OID of the empty tree. */
>         const struct object_id *empty_tree;
>     @@ object-file.c: static void git_hash_sha1_final_oid(struct object_id *oid, git_ha
>         oid->algo = GIT_HASH_SHA1;
>       }
>
>     -+static void git_hash_sha1_init_fast(git_hash_ctx *ctx)
>     ++static void git_hash_sha1_init_unsafe(git_hash_ctx *ctx)
>      +{
>     -+  git_SHA1_Init_fast(&ctx->sha1_fast);
>     ++  git_SHA1_Init_unsafe(&ctx->sha1_unsafe);
>      +}
>      +
>     -+static void git_hash_sha1_clone_fast(git_hash_ctx *dst, const git_hash_ctx *src)
>     ++static void git_hash_sha1_clone_unsafe(git_hash_ctx *dst, const git_hash_ctx *src)
>      +{
>     -+  git_SHA1_Clone_fast(&dst->sha1_fast, &src->sha1_fast);
>     ++  git_SHA1_Clone_unsafe(&dst->sha1_unsafe, &src->sha1_unsafe);
>      +}
>      +
>     -+static void git_hash_sha1_update_fast(git_hash_ctx *ctx, const void *data,
>     ++static void git_hash_sha1_update_unsafe(git_hash_ctx *ctx, const void *data,
>      +                                size_t len)
>      +{
>     -+  git_SHA1_Update_fast(&ctx->sha1_fast, data, len);
>     ++  git_SHA1_Update_unsafe(&ctx->sha1_unsafe, data, len);
>      +}
>      +
>     -+static void git_hash_sha1_final_fast(unsigned char *hash, git_hash_ctx *ctx)
>     ++static void git_hash_sha1_final_unsafe(unsigned char *hash, git_hash_ctx *ctx)
>      +{
>     -+  git_SHA1_Final_fast(hash, &ctx->sha1_fast);
>     ++  git_SHA1_Final_unsafe(hash, &ctx->sha1_unsafe);
>      +}
>      +
>     -+static void git_hash_sha1_final_oid_fast(struct object_id *oid, git_hash_ctx *ctx)
>     ++static void git_hash_sha1_final_oid_unsafe(struct object_id *oid, git_hash_ctx *ctx)
>      +{
>     -+  git_SHA1_Final_fast(oid->hash, &ctx->sha1_fast);
>     ++  git_SHA1_Final_unsafe(oid->hash, &ctx->sha1_unsafe);
>      +  memset(oid->hash + GIT_SHA1_RAWSZ, 0, GIT_MAX_RAWSZ - GIT_SHA1_RAWSZ);
>      +  oid->algo = GIT_HASH_SHA1;
>      +}
>     @@ object-file.c: const struct git_hash_algo hash_algos[GIT_HASH_NALGOS] = {
>                 .update_fn = git_hash_unknown_update,
>                 .final_fn = git_hash_unknown_final,
>                 .final_oid_fn = git_hash_unknown_final_oid,
>     -+          .fast_init_fn = git_hash_unknown_init,
>     -+          .fast_clone_fn = git_hash_unknown_clone,
>     -+          .fast_update_fn = git_hash_unknown_update,
>     -+          .fast_final_fn = git_hash_unknown_final,
>     -+          .fast_final_oid_fn = git_hash_unknown_final_oid,
>     ++          .unsafe_init_fn = git_hash_unknown_init,
>     ++          .unsafe_clone_fn = git_hash_unknown_clone,
>     ++          .unsafe_update_fn = git_hash_unknown_update,
>     ++          .unsafe_final_fn = git_hash_unknown_final,
>     ++          .unsafe_final_oid_fn = git_hash_unknown_final_oid,
>                 .empty_tree = NULL,
>                 .empty_blob = NULL,
>                 .null_oid = NULL,
>     @@ object-file.c: const struct git_hash_algo hash_algos[GIT_HASH_NALGOS] = {
>                 .update_fn = git_hash_sha1_update,
>                 .final_fn = git_hash_sha1_final,
>                 .final_oid_fn = git_hash_sha1_final_oid,
>     -+          .fast_init_fn = git_hash_sha1_init_fast,
>     -+          .fast_clone_fn = git_hash_sha1_clone_fast,
>     -+          .fast_update_fn = git_hash_sha1_update_fast,
>     -+          .fast_final_fn = git_hash_sha1_final_fast,
>     -+          .fast_final_oid_fn = git_hash_sha1_final_oid_fast,
>     ++          .unsafe_init_fn = git_hash_sha1_init_unsafe,
>     ++          .unsafe_clone_fn = git_hash_sha1_clone_unsafe,
>     ++          .unsafe_update_fn = git_hash_sha1_update_unsafe,
>     ++          .unsafe_final_fn = git_hash_sha1_final_unsafe,
>     ++          .unsafe_final_oid_fn = git_hash_sha1_final_oid_unsafe,
>                 .empty_tree = &empty_tree_oid,
>                 .empty_blob = &empty_blob_oid,
>                 .null_oid = &null_oid_sha1,
>     @@ object-file.c: const struct git_hash_algo hash_algos[GIT_HASH_NALGOS] = {
>                 .update_fn = git_hash_sha256_update,
>                 .final_fn = git_hash_sha256_final,
>                 .final_oid_fn = git_hash_sha256_final_oid,
>     -+          .fast_init_fn = git_hash_sha256_init,
>     -+          .fast_clone_fn = git_hash_sha256_clone,
>     -+          .fast_update_fn = git_hash_sha256_update,
>     -+          .fast_final_fn = git_hash_sha256_final,
>     -+          .fast_final_oid_fn = git_hash_sha256_final_oid,
>     ++          .unsafe_init_fn = git_hash_sha256_init,
>     ++          .unsafe_clone_fn = git_hash_sha256_clone,
>     ++          .unsafe_update_fn = git_hash_sha256_update,
>     ++          .unsafe_final_fn = git_hash_sha256_final,
>     ++          .unsafe_final_oid_fn = git_hash_sha256_final_oid,
>                 .empty_tree = &empty_tree_oid_sha256,
>                 .empty_blob = &empty_blob_oid_sha256,
>                 .null_oid = &null_oid_sha256,

This is basically a big s/fast/unsafe/ replace.  Simple enough.

>  8:  137ec30d68a !  7:  af8fd9aa4ed Makefile: allow specifying a SHA-1 for non-cryptographic uses
>     @@ Metadata
>       ## Commit message ##
>          Makefile: allow specifying a SHA-1 for non-cryptographic uses
>
>     -    Introduce _FAST variants of the OPENSSL_SHA1, BLK_SHA1, and
>     +    Introduce _UNSAFE variants of the OPENSSL_SHA1, BLK_SHA1, and
>          APPLE_COMMON_CRYPTO_SHA1 compile-time knobs which indicate which SHA-1
>          implementation is to be used for non-cryptographic uses.
>
>     @@ Makefile: include shared.mak
>       # Define APPLE_COMMON_CRYPTO_SHA1 to use Apple's CommonCrypto for
>       # SHA-1.
>       #
>     -+# Define the same Makefile knobs as above, but suffixed with _FAST to
>     -+# use the corresponding implementations for "fast" SHA-1 hashing for
>     ++# Define the same Makefile knobs as above, but suffixed with _UNSAFE to
>     ++# use the corresponding implementations for unsafe SHA-1 hashing for
>      +# non-cryptographic purposes.
>      +#
>       # If don't enable any of the *_SHA1 settings in this section, Git will
>     @@ Makefile: endif
>       endif
>       endif
>
>     -+ifdef OPENSSL_SHA1_FAST
>     ++ifdef OPENSSL_SHA1_UNSAFE
>      +ifndef OPENSSL_SHA1
>      +  EXTLIBS += $(LIB_4_CRYPTO)
>     -+  BASIC_CFLAGS += -DSHA1_OPENSSL_FAST
>     ++  BASIC_CFLAGS += -DSHA1_OPENSSL_UNSAFE
>      +endif
>      +else
>     -+ifdef BLK_SHA1_FAST
>     ++ifdef BLK_SHA1_UNSAFE
>      +ifndef BLK_SHA1
>      +  LIB_OBJS += block-sha1/sha1.o
>     -+  BASIC_CFLAGS += -DSHA1_BLK_FAST
>     ++  BASIC_CFLAGS += -DSHA1_BLK_UNSAFE
>      +endif
>      +else
>     -+ifdef APPLE_COMMON_CRYPTO_SHA1_FAST
>     ++ifdef APPLE_COMMON_CRYPTO_SHA1_UNSAFE
>      +ifndef APPLE_COMMON_CRYPTO_SHA1
>      +  COMPAT_CFLAGS += -DCOMMON_DIGEST_FOR_OPENSSL
>     -+  BASIC_CFLAGS += -DSHA1_APPLE_FAST
>     ++  BASIC_CFLAGS += -DSHA1_APPLE_UNSAFE
>      +endif
>      +endif
>      +endif
>     @@ hash.h
>       #include "block-sha1/sha1.h"
>       #endif
>
>     -+#if defined(SHA1_APPLE_FAST)
>     ++#if defined(SHA1_APPLE_UNSAFE)
>      +#  include <CommonCrypto/CommonDigest.h>
>     -+#  define platform_SHA_CTX_fast CC_SHA1_CTX
>     -+#  define platform_SHA1_Init_fast CC_SHA1_Init
>     -+#  define platform_SHA1_Update_fast CC_SHA1_Update
>     -+#  define platform_SHA1_Final_fast CC_SHA1_Final
>     -+#elif defined(SHA1_OPENSSL_FAST)
>     ++#  define platform_SHA_CTX_unsafe CC_SHA1_CTX
>     ++#  define platform_SHA1_Init_unsafe CC_SHA1_Init
>     ++#  define platform_SHA1_Update_unsafe CC_SHA1_Update
>     ++#  define platform_SHA1_Final_unsafe CC_SHA1_Final
>     ++#elif defined(SHA1_OPENSSL_UNSAFE)
>      +#  include <openssl/sha.h>
>      +#  if defined(OPENSSL_API_LEVEL) && OPENSSL_API_LEVEL >= 3
>     -+#    define SHA1_NEEDS_CLONE_HELPER_FAST
>     ++#    define SHA1_NEEDS_CLONE_HELPER_UNSAFE
>      +#    include "sha1/openssl.h"
>     -+#    define platform_SHA_CTX_fast openssl_SHA1_CTX
>     -+#    define platform_SHA1_Init_fast openssl_SHA1_Init
>     -+#    define platform_SHA1_Clone_fast openssl_SHA1_Clone
>     -+#    define platform_SHA1_Update_fast openssl_SHA1_Update
>     -+#    define platform_SHA1_Final_fast openssl_SHA1_Final
>     ++#    define platform_SHA_CTX_unsafe openssl_SHA1_CTX
>     ++#    define platform_SHA1_Init_unsafe openssl_SHA1_Init
>     ++#    define platform_SHA1_Clone_unsafe openssl_SHA1_Clone
>     ++#    define platform_SHA1_Update_unsafe openssl_SHA1_Update
>     ++#    define platform_SHA1_Final_unsafe openssl_SHA1_Final
>      +#  else
>     -+#    define platform_SHA_CTX_fast SHA_CTX
>     -+#    define platform_SHA1_Init_fast SHA1_Init
>     -+#    define platform_SHA1_Update_fast SHA1_Update
>     -+#    define platform_SHA1_Final_fast SHA1_Final
>     ++#    define platform_SHA_CTX_unsafe SHA_CTX
>     ++#    define platform_SHA1_Init_unsafe SHA1_Init
>     ++#    define platform_SHA1_Update_unsafe SHA1_Update
>     ++#    define platform_SHA1_Final_unsafe SHA1_Final
>      +#  endif
>     -+#elif defined(SHA1_BLK_FAST)
>     ++#elif defined(SHA1_BLK_UNSAFE)
>      +#  include "block-sha1/sha1.h"
>     -+#  define platform_SHA_CTX_fast blk_SHA_CTX
>     -+#  define platform_SHA1_Init_fast blk_SHA1_Init
>     -+#  define platform_SHA1_Update_fast blk_SHA1_Update
>     -+#  define platform_SHA1_Final_fast blk_SHA1_Final
>     ++#  define platform_SHA_CTX_unsafe blk_SHA_CTX
>     ++#  define platform_SHA1_Init_unsafe blk_SHA1_Init
>     ++#  define platform_SHA1_Update_unsafe blk_SHA1_Update
>     ++#  define platform_SHA1_Final_unsafe blk_SHA1_Final
>      +#endif
>      +

Likewise.

>       #if defined(SHA256_NETTLE)
>  9:  4018261366f !  8:  4b83dd05e9f csum-file.c: use fast SHA-1 implementation when available
>     @@ Metadata
>      Author: Taylor Blau <me@ttaylorr.com>
>
>       ## Commit message ##
>     -    csum-file.c: use fast SHA-1 implementation when available
>     +    csum-file.c: use unsafe SHA-1 implementation when available
>
>     -    Update hashwrite() and friends to use the fast_-variants of hashing
>     -    functions, calling for e.g., "the_hash_algo->fast_update_fn()" instead
>     +    Update hashwrite() and friends to use the unsafe_-variants of hashing
>     +    functions, calling for e.g., "the_hash_algo->unsafe_update_fn()" instead
>          of "the_hash_algo->update_fn()".
>
>          These callers only use the_hash_algo to produce a checksum, which we
>          depend on for data integrity, but not for cryptographic purposes, so
>     -    these callers are safe to use the fast (and potentially non-collision
>     +    these callers are safe to use the unsafe (and potentially non-collision
>          detecting) SHA-1 implementation.

Is the "and potentially non-collision detecting" parenthetical comment
still needed now that it's referred to as unsafe?  Even if we keep
most of it, maybe we should drop the "and"?

>
>          To time this, I took a freshly packed copy of linux.git, and ran the
>     -    following with and without the OPENSSL_SHA1_FAST=1 build-knob. Both
>     +    following with and without the OPENSSL_SHA1_UNSAFE=1 build-knob. Both
>          versions were compiled with -O3:
>
>              $ git for-each-ref --format='%(objectname)' refs/heads refs/tags >in
>              $ valgrind --tool=callgrind ~/src/git/git-pack-objects \
>                  --revs --stdout --all-progress --use-bitmap-index <in >/dev/null
>
>     -    Without OPENSSL_SHA1_FAST=1 (that is, using the collision-detecting
>     +    Without OPENSSL_SHA1_UNSAFE=1 (that is, using the collision-detecting
>          SHA-1 implementation for both cryptographic and non-cryptographic
>          purposes), we spend a significant amount of our instruction count in
>          hashwrite():
>     @@ Commit message
>          , and the resulting "clone" takes 19.219 seconds of wall clock time,
>          18.94 seconds of user time and 0.28 seconds of system time.
>
>     -    Compiling with OPENSSL_SHA1_FAST=1, we spend ~60% fewer instructions in
>     -    hashwrite():
>     +    Compiling with OPENSSL_SHA1_UNSAFE=1, we spend ~60% fewer instructions
>     +    in hashwrite():
>
>              $ callgrind_annotate --inclusive=yes | grep hashwrite | head -n1
>               59,164,001,176 (58.79%)  /home/ttaylorr/src/git/csum-file.c:hashwrite [/home/ttaylorr/src/git/git-pack-objects]
>
>     -    , and generate the resulting "clone" much faster, in only 11.597 seconds
>     +    , and generate the resulting "clone" much unsafeer, in only 11.597 seconds

This fast->unsafe translation isn't so good; this one should be undone.

>          of wall time, 11.37 seconds of user time, and 0.23 seconds of system
>          time, for a ~40% speed-up.
>
>     @@ csum-file.c: void hashflush(struct hashfile *f)
>         if (offset) {
>                 if (!f->skip_hash)
>      -                  the_hash_algo->update_fn(&f->ctx, f->buffer, offset);
>     -+                  the_hash_algo->fast_update_fn(&f->ctx, f->buffer, offset);
>     ++                  the_hash_algo->unsafe_update_fn(&f->ctx, f->buffer, offset);
>                 flush(f, f->buffer, offset);
>                 f->offset = 0;
>         }
>     @@ csum-file.c: int finalize_hashfile(struct hashfile *f, unsigned char *result,
>                 hashclr(f->buffer, the_repository->hash_algo);
>         else
>      -          the_hash_algo->final_fn(f->buffer, &f->ctx);
>     -+          the_hash_algo->fast_final_fn(f->buffer, &f->ctx);
>     ++          the_hash_algo->unsafe_final_fn(f->buffer, &f->ctx);
>
>         if (result)
>                 hashcpy(result, f->buffer, the_repository->hash_algo);
>     @@ csum-file.c: void hashwrite(struct hashfile *f, const void *buf, unsigned int co
>                          */
>                         if (!f->skip_hash)
>      -                          the_hash_algo->update_fn(&f->ctx, buf, nr);
>     -+                          the_hash_algo->fast_update_fn(&f->ctx, buf, nr);
>     ++                          the_hash_algo->unsafe_update_fn(&f->ctx, buf, nr);
>                         flush(f, buf, nr);
>                 } else {
>                         /*
>     @@ csum-file.c: static struct hashfile *hashfd_internal(int fd, const char *name,
>         f->do_crc = 0;
>         f->skip_hash = 0;
>      -  the_hash_algo->init_fn(&f->ctx);
>     -+  the_hash_algo->fast_init_fn(&f->ctx);
>     ++  the_hash_algo->unsafe_init_fn(&f->ctx);
>
>         f->buffer_len = buffer_len;
>         f->buffer = xmalloc(buffer_len);
>     @@ csum-file.c: void hashfile_checkpoint(struct hashfile *f, struct hashfile_checkp
>         hashflush(f);
>         checkpoint->offset = f->total;
>      -  the_hash_algo->clone_fn(&checkpoint->ctx, &f->ctx);
>     -+  the_hash_algo->fast_clone_fn(&checkpoint->ctx, &f->ctx);
>     ++  the_hash_algo->unsafe_clone_fn(&checkpoint->ctx, &f->ctx);
>       }
>
>       int hashfile_truncate(struct hashfile *f, struct hashfile_checkpoint *checkpoint)
>     @@ csum-file.c: int hashfile_truncate(struct hashfile *f, struct hashfile_checkpoin
>                 return -1;
>         f->total = offset;
>      -  the_hash_algo->clone_fn(&f->ctx, &checkpoint->ctx);
>     -+  the_hash_algo->fast_clone_fn(&f->ctx, &checkpoint->ctx);
>     ++  the_hash_algo->unsafe_clone_fn(&f->ctx, &checkpoint->ctx);
>         f->offset = 0; /* hashflush() was called in checkpoint */
>         return 0;
>       }
>     @@ csum-file.c: int hashfile_checksum_valid(const unsigned char *data, size_t total
>      -  the_hash_algo->init_fn(&ctx);
>      -  the_hash_algo->update_fn(&ctx, data, data_len);
>      -  the_hash_algo->final_fn(got, &ctx);
>     -+  the_hash_algo->fast_init_fn(&ctx);
>     -+  the_hash_algo->fast_update_fn(&ctx, data, data_len);
>     -+  the_hash_algo->fast_final_fn(got, &ctx);
>     ++  the_hash_algo->unsafe_init_fn(&ctx);
>     ++  the_hash_algo->unsafe_update_fn(&ctx, data, data_len);
>     ++  the_hash_algo->unsafe_final_fn(got, &ctx);
>
>         return hasheq(got, data + data_len, the_repository->hash_algo);
>       }

This patch was also fast->unsafe translations; I identified two above
that I think should get some tweaks.

Re: [PATCH v4 0/8] hash.h: support choosing a separate SHA-1 for non-cryptographic uses

[Junio C Hamano]

Elijah Newren <newren@gmail.com> writes:

>>     -    these callers are safe to use the fast (and potentially non-collision
>>     +    these callers are safe to use the unsafe (and potentially non-collision
>>          detecting) SHA-1 implementation.
>
> Is the "and potentially non-collision detecting" parenthetical comment
> still needed now that it's referred to as unsafe?  Even if we keep
> most of it, maybe we should drop the "and"?

I appreciate a careful reading like this.  The use of "unsafe"
becomes easier to understandable if we lost "potentially", e.g.

	are safe to use the unsafe SHA-1 implementation that does
	not attempt to detect collisions.

>>     -    , and generate the resulting "clone" much faster, in only 11.597 seconds
>>     +    , and generate the resulting "clone" much unsafeer, in only 11.597 seconds
>
> This fast->unsafe translation isn't so good; this one should be undone.

Or "much less safe", but that is not something we want to brag about ;-)

> ...
> This patch was also fast->unsafe translations; I identified two above
> that I think should get some tweaks.

Thanks for carefully reading.

Re: [PATCH v4 0/8] hash.h: support choosing a separate SHA-1 for non-cryptographic uses

[Taylor Blau]

On Wed, Sep 25, 2024 at 10:11:20AM -0700, Junio C Hamano wrote:
> Elijah Newren <newren@gmail.com> writes:
>
> >>     -    these callers are safe to use the fast (and potentially non-collision
> >>     +    these callers are safe to use the unsafe (and potentially non-collision
> >>          detecting) SHA-1 implementation.
> >
> > Is the "and potentially non-collision detecting" parenthetical comment
> > still needed now that it's referred to as unsafe?  Even if we keep
> > most of it, maybe we should drop the "and"?
>
> I appreciate a careful reading like this.  The use of "unsafe"
> becomes easier to understandable if we lost "potentially", e.g.
>
> 	are safe to use the unsafe SHA-1 implementation that does
> 	not attempt to detect collisions.

Me too. I ended up with a slightly different wording that matches
Elijah's more than your own, but I appreciate the careful review from
both of you.

> > This patch was also fast->unsafe translations; I identified two above
> > that I think should get some tweaks.
>
> Thanks for carefully reading.

Ditto.

Thanks,
Taylor

Re: [PATCH v4 0/8] hash.h: support choosing a separate SHA-1 for non-cryptographic uses

[Taylor Blau]

On Wed, Sep 25, 2024 at 09:58:47AM -0700, Elijah Newren wrote:
> >          Co-authored-by: Jeff King <peff@peff.net>
> >          Signed-off-by: Jeff King <peff@peff.net>
>
> I don't even get a Helped-by for finding the cause of the pile-up of
> temporary object files and the suggested tweak to this patch?  ;-)

Oops, that is definitely an oversight. I'll add your Helped-by to the
next version (though it really should probably be a
My-bacon-was-saved-by trailer ;-)).

> >  9:  4018261366f !  8:  4b83dd05e9f csum-file.c: use fast SHA-1 implementation when available
> >     @@ Metadata
> >      Author: Taylor Blau <me@ttaylorr.com>
> >
> >       ## Commit message ##
> >     -    csum-file.c: use fast SHA-1 implementation when available
> >     +    csum-file.c: use unsafe SHA-1 implementation when available
> >
> >     -    Update hashwrite() and friends to use the fast_-variants of hashing
> >     -    functions, calling for e.g., "the_hash_algo->fast_update_fn()" instead
> >     +    Update hashwrite() and friends to use the unsafe_-variants of hashing
> >     +    functions, calling for e.g., "the_hash_algo->unsafe_update_fn()" instead
> >          of "the_hash_algo->update_fn()".
> >
> >          These callers only use the_hash_algo to produce a checksum, which we
> >          depend on for data integrity, but not for cryptographic purposes, so
> >     -    these callers are safe to use the fast (and potentially non-collision
> >     +    these callers are safe to use the unsafe (and potentially non-collision
> >          detecting) SHA-1 implementation.
>
> Is the "and potentially non-collision detecting" parenthetical comment
> still needed now that it's referred to as unsafe?  Even if we keep
> most of it, maybe we should drop the "and"?

Yeah, I think that's a good idea. Without specifying any of the new
build knobs, the "unsafe" function pointers are identical to the safe
ones, which is why I wrote "potentially" here. But I think that's a
semantic argument that is confusing at best and misleading at worst, so
I like your suggestion to make the parenthetical just read
"(non-collision detecting)".

> >     @@ Commit message
> >          , and the resulting "clone" takes 19.219 seconds of wall clock time,
> >          18.94 seconds of user time and 0.28 seconds of system time.
> >
> >     -    Compiling with OPENSSL_SHA1_FAST=1, we spend ~60% fewer instructions in
> >     -    hashwrite():
> >     +    Compiling with OPENSSL_SHA1_UNSAFE=1, we spend ~60% fewer instructions
> >     +    in hashwrite():
> >
> >              $ callgrind_annotate --inclusive=yes | grep hashwrite | head -n1
> >               59,164,001,176 (58.79%)  /home/ttaylorr/src/git/csum-file.c:hashwrite [/home/ttaylorr/src/git/git-pack-objects]
> >
> >     -    , and generate the resulting "clone" much faster, in only 11.597 seconds
> >     +    , and generate the resulting "clone" much unsafeer, in only 11.597 seconds
>
> This fast->unsafe translation isn't so good; this one should be undone.

Oops. Great catch, thanks.

Thanks,
Taylor

[PATCH v5 0/8] hash.h: support choosing a separate SHA-1 for non-cryptographic uses

[Taylor Blau]

Here is a minor reroll of mine and Peff's series to add a build-time
knob to allow selecting an alternative SHA-1 implementation for
non-cryptographic hashing within Git, starting with the `hashwrite()`
family of functions.

This version has changes which are limited to the commit messages only,
and amount to:

  - Updated rationale for skipping the collision check from within
    finalize_object_file() when handling loose objects.

  - Updated commit message with some over-eager s/fast/unsafe/
    conversions in the final patch.

I think most of the review dust has settled up to this point, so I'm
imagining that this is the final version of this series for now, or at
least very close to it. But if something new comes up, please let me
know!

Thanks in advance for your review!

Taylor Blau (8):
  finalize_object_file(): check for name collision before renaming
  finalize_object_file(): refactor unlink_or_warn() placement
  finalize_object_file(): implement collision check
  pack-objects: use finalize_object_file() to rename pack/idx/etc
  sha1: do not redefine `platform_SHA_CTX` and friends
  hash.h: scaffolding for _unsafe hashing variants
  Makefile: allow specifying a SHA-1 for non-cryptographic uses
  csum-file.c: use unsafe SHA-1 implementation when available

 Makefile                              |  25 ++++++
 block-sha1/sha1.h                     |   2 +
 csum-file.c                           |  18 ++--
 hash.h                                |  72 +++++++++++++++
 object-file.c                         | 124 ++++++++++++++++++++++++--
 object-file.h                         |   6 ++
 pack-write.c                          |   7 +-
 sha1/openssl.h                        |   2 +
 sha1dc_git.h                          |   3 +
 t/t5303-pack-corruption-resilience.sh |   7 +-
 tmp-objdir.c                          |  26 ++++--
 11 files changed, 266 insertions(+), 26 deletions(-)

Range-diff against v4:
-:  ---------- > 1:  6f1ee91fff finalize_object_file(): check for name collision before renaming
-:  ---------- > 2:  133047ca8c finalize_object_file(): refactor unlink_or_warn() placement
1:  ed9eeef851 ! 3:  41d38352a4 finalize_object_file(): implement collision check
    @@ Commit message
             object name, so checking for collisions at the content level doesn't
             add anything.
     
    -        This is why we do not bother to check the inflated object contents
    -        for collisions either, since either (a) the object contents have the
    -        fingerprint of a SHA-1 collision, in which case the collision
    -        detecting SHA-1 implementation used to hash the contents to give us
    -        a path would have already rejected it, or (b) the contents are part
    -        of a colliding pair which does not bear the same fingerprints of
    -        known collision attacks, in which case we would not have caught it
    -        anyway.
    +        Adding a content-level collision check would have to happen at a
    +        higher level than in finalize_object_file(), since (avoiding race
    +        conditions) writing an object loose which already exists in the
    +        repository will prevent us from even reaching finalize_object_file()
    +        via the object freshening code.
    +
    +        There is a collision check in index-pack via its `check_collision()`
    +        function, but there isn't an analogous function in unpack-objects,
    +        which just feeds the result to write_object_file().
     
             So skipping the collision check here does not change for better or
             worse the hardness of loose object writes.
    @@ Commit message
     
         Co-authored-by: Jeff King <peff@peff.net>
         Signed-off-by: Jeff King <peff@peff.net>
    +    Helped-by: Elijah Newren <newren@gmail.com>
         Signed-off-by: Taylor Blau <me@ttaylorr.com>
     
      ## object-file.c ##
2:  3cc7f7b1f6 = 4:  611475d83e pack-objects: use finalize_object_file() to rename pack/idx/etc
3:  8f8ac0f5b0 = 5:  9913a5d971 sha1: do not redefine `platform_SHA_CTX` and friends
4:  d300e9c688 = 6:  65de6d724d hash.h: scaffolding for _unsafe hashing variants
5:  af8fd9aa4e = 7:  3884cd0e3a Makefile: allow specifying a SHA-1 for non-cryptographic uses
6:  4b83dd05e9 ! 8:  62abddf73d csum-file.c: use unsafe SHA-1 implementation when available
    @@ Commit message
     
         These callers only use the_hash_algo to produce a checksum, which we
         depend on for data integrity, but not for cryptographic purposes, so
    -    these callers are safe to use the unsafe (and potentially non-collision
    -    detecting) SHA-1 implementation.
    +    these callers are safe to use the unsafe (non-collision detecting) SHA-1
    +    implementation.
     
         To time this, I took a freshly packed copy of linux.git, and ran the
         following with and without the OPENSSL_SHA1_UNSAFE=1 build-knob. Both
    @@ Commit message
             $ callgrind_annotate --inclusive=yes | grep hashwrite | head -n1
              59,164,001,176 (58.79%)  /home/ttaylorr/src/git/csum-file.c:hashwrite [/home/ttaylorr/src/git/git-pack-objects]
     
    -    , and generate the resulting "clone" much unsafeer, in only 11.597 seconds
    +    , and generate the resulting "clone" much faster, in only 11.597 seconds
         of wall time, 11.37 seconds of user time, and 0.23 seconds of system
         time, for a ~40% speed-up.
     

base-commit: 6258f68c3c1092c901337895c864073dcdea9213
-- 
2.46.1.507.gffd0c9a15b2.dirty

[PATCH v5 1/8] finalize_object_file(): check for name collision before renaming

[Taylor Blau]

We prefer link()/unlink() to rename() for object files, with the idea
that we should prefer the data that is already on disk to what is
incoming. But we may fall back to rename() if the user has configured us
to do so, or if the filesystem seems not to support cross-directory
links. This loses the "prefer what is on disk" property.

We can mitigate this somewhat by trying to stat() the destination
filename before doing the rename. This is racy, since the object could
be created between the stat() and rename() calls. But in practice it is
expanding the definition of "what is already on disk" to be the point
that the function is called. That is enough to deal with any potential
attacks where an attacker is trying to collide hashes with what's
already in the repository.

Co-authored-by: Jeff King <peff@peff.net>
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Taylor Blau <me@ttaylorr.com>
---
 object-file.c | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/object-file.c b/object-file.c
index 968da27cd4..38407f468a 100644
--- a/object-file.c
+++ b/object-file.c
@@ -1937,6 +1937,7 @@ static void write_object_file_prepare_literally(const struct git_hash_algo *algo
  */
 int finalize_object_file(const char *tmpfile, const char *filename)
 {
+	struct stat st;
 	int ret = 0;
 
 	if (object_creation_mode == OBJECT_CREATION_USES_RENAMES)
@@ -1957,9 +1958,12 @@ int finalize_object_file(const char *tmpfile, const char *filename)
 	 */
 	if (ret && ret != EEXIST) {
 	try_rename:
-		if (!rename(tmpfile, filename))
+		if (!stat(filename, &st))
+			ret = EEXIST;
+		else if (!rename(tmpfile, filename))
 			goto out;
-		ret = errno;
+		else
+			ret = errno;
 	}
 	unlink_or_warn(tmpfile);
 	if (ret) {
-- 
2.46.1.507.gffd0c9a15b2.dirty

[PATCH v5 2/8] finalize_object_file(): refactor unlink_or_warn() placement

[Taylor Blau]

As soon as we've tried to link() a temporary object into place, we then
unlink() the tempfile immediately, whether we were successful or not.

For the success case, this is because we no longer need the old file
(it's now linked into place).

For the error case, there are two outcomes. Either we got EEXIST, in
which case we consider the collision to be a noop. Or we got a system
error, in which we case we are just cleaning up after ourselves.

Using a single line for all of these cases has some problems:

  - in the error case, our unlink() may clobber errno, which we use in
    the error message

  - for the collision case, there's a FIXME that indicates we should do
    a collision check. In preparation for implementing that, we'll need
    to actually hold on to the file.

Split these three cases into their own calls to unlink_or_warn(). This
is more verbose, but lets us do the right thing in each case.

Co-authored-by: Jeff King <peff@peff.net>
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Taylor Blau <me@ttaylorr.com>
---
 object-file.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/object-file.c b/object-file.c
index 38407f468a..5a1da57711 100644
--- a/object-file.c
+++ b/object-file.c
@@ -1944,6 +1944,8 @@ int finalize_object_file(const char *tmpfile, const char *filename)
 		goto try_rename;
 	else if (link(tmpfile, filename))
 		ret = errno;
+	else
+		unlink_or_warn(tmpfile);
 
 	/*
 	 * Coda hack - coda doesn't like cross-directory links,
@@ -1965,12 +1967,15 @@ int finalize_object_file(const char *tmpfile, const char *filename)
 		else
 			ret = errno;
 	}
-	unlink_or_warn(tmpfile);
 	if (ret) {
 		if (ret != EEXIST) {
+			int saved_errno = errno;
+			unlink_or_warn(tmpfile);
+			errno = saved_errno;
 			return error_errno(_("unable to write file %s"), filename);
 		}
 		/* FIXME!!! Collision check here ? */
+		unlink_or_warn(tmpfile);
 	}
 
 out:
-- 
2.46.1.507.gffd0c9a15b2.dirty

[PATCH v5 3/8] finalize_object_file(): implement collision check

[Taylor Blau]

We've had "FIXME!!! Collision check here ?" in finalize_object_file()
since aac1794132 (Improve sha1 object file writing., 2005-05-03). That
is, when we try to write a file with the same name, we assume the
on-disk contents are the same and blindly throw away the new copy.

One of the reasons we never implemented this is because the files it
moves are all named after the cryptographic hash of their contents
(either loose objects, or packs which have their hash in the name these
days). So we are unlikely to see such a collision by accident. And even
though there are weaknesses in sha1, we assume they are mitigated by our
use of sha1dc.

So while it's a theoretical concern now, it hasn't been a priority.
However, if we start using weaker hashes for pack checksums and names,
this will become a practical concern. So in preparation, let's actually
implement a byte-for-byte collision check.

The new check will cause the write of new differing content to be a
failure, rather than a silent noop, and we'll retain the temporary file
on disk. If there's no collision present, we'll clean up the temporary
file as usual after either rename()-ing or link()-ing it into place.

Note that this may cause some extra computation when the files are in
fact identical, but this should happen rarely.

Loose objects are exempt from this check, and the collision check may be
skipped by calling the _flags variant of this function with the
FOF_SKIP_COLLISION_CHECK bit set. This is done for a couple of reasons:

  - We don't treat the hash of the loose object file's contents as a
    checksum, since the same loose object can be stored using different
    bytes on disk (e.g., when adjusting core.compression, using a
    different version of zlib, etc.).

    This is fundamentally different from cases where
    finalize_object_file() is operating over a file which uses the hash
    value as a checksum of the contents. In other words, a pair of
    identical loose objects can be stored using different bytes on disk,
    and that should not be treated as a collision.

  - We already use the path of the loose object as its hash value /
    object name, so checking for collisions at the content level doesn't
    add anything.

    Adding a content-level collision check would have to happen at a
    higher level than in finalize_object_file(), since (avoiding race
    conditions) writing an object loose which already exists in the
    repository will prevent us from even reaching finalize_object_file()
    via the object freshening code.

    There is a collision check in index-pack via its `check_collision()`
    function, but there isn't an analogous function in unpack-objects,
    which just feeds the result to write_object_file().

    So skipping the collision check here does not change for better or
    worse the hardness of loose object writes.

As a small note related to the latter bullet point above, we must teach
the tmp-objdir routines to similarly skip the content-level collision
checks when calling migrate_one() on a loose object file, which we do by
setting the FOF_SKIP_COLLISION_CHECK bit when we are inside of a loose
object shard.

Co-authored-by: Jeff King <peff@peff.net>
Signed-off-by: Jeff King <peff@peff.net>
Helped-by: Elijah Newren <newren@gmail.com>
Signed-off-by: Taylor Blau <me@ttaylorr.com>
---
 object-file.c | 67 ++++++++++++++++++++++++++++++++++++++++++++++++---
 object-file.h |  6 +++++
 tmp-objdir.c  | 26 ++++++++++++++------
 3 files changed, 89 insertions(+), 10 deletions(-)

diff --git a/object-file.c b/object-file.c
index 5a1da57711..b9a3a1f62d 100644
--- a/object-file.c
+++ b/object-file.c
@@ -1932,10 +1932,67 @@ static void write_object_file_prepare_literally(const struct git_hash_algo *algo
 	hash_object_body(algo, &c, buf, len, oid, hdr, hdrlen);
 }
 
+static int check_collision(const char *filename_a, const char *filename_b)
+{
+	char buf_a[4096], buf_b[4096];
+	int fd_a = -1, fd_b = -1;
+	int ret = 0;
+
+	fd_a = open(filename_a, O_RDONLY);
+	if (fd_a < 0) {
+		ret = error_errno(_("unable to open %s"), filename_a);
+		goto out;
+	}
+
+	fd_b = open(filename_b, O_RDONLY);
+	if (fd_b < 0) {
+		ret = error_errno(_("unable to open %s"), filename_b);
+		goto out;
+	}
+
+	while (1) {
+		ssize_t sz_a, sz_b;
+
+		sz_a = read_in_full(fd_a, buf_a, sizeof(buf_a));
+		if (sz_a < 0) {
+			ret = error_errno(_("unable to read %s"), filename_a);
+			goto out;
+		}
+
+		sz_b = read_in_full(fd_b, buf_b, sizeof(buf_b));
+		if (sz_b < 0) {
+			ret = error_errno(_("unable to read %s"), filename_b);
+			goto out;
+		}
+
+		if (sz_a != sz_b || memcmp(buf_a, buf_b, sz_a)) {
+			ret = error(_("files '%s' and '%s' differ in contents"),
+				    filename_a, filename_b);
+			goto out;
+		}
+
+		if (sz_a < sizeof(buf_a))
+			break;
+	}
+
+out:
+	if (fd_a > -1)
+		close(fd_a);
+	if (fd_b > -1)
+		close(fd_b);
+	return ret;
+}
+
 /*
  * Move the just written object into its final resting place.
  */
 int finalize_object_file(const char *tmpfile, const char *filename)
+{
+	return finalize_object_file_flags(tmpfile, filename, 0);
+}
+
+int finalize_object_file_flags(const char *tmpfile, const char *filename,
+			       enum finalize_object_file_flags flags)
 {
 	struct stat st;
 	int ret = 0;
@@ -1974,7 +2031,9 @@ int finalize_object_file(const char *tmpfile, const char *filename)
 			errno = saved_errno;
 			return error_errno(_("unable to write file %s"), filename);
 		}
-		/* FIXME!!! Collision check here ? */
+		if (!(flags & FOF_SKIP_COLLISION_CHECK) &&
+		    check_collision(tmpfile, filename))
+				return -1;
 		unlink_or_warn(tmpfile);
 	}
 
@@ -2228,7 +2287,8 @@ static int write_loose_object(const struct object_id *oid, char *hdr,
 			warning_errno(_("failed utime() on %s"), tmp_file.buf);
 	}
 
-	return finalize_object_file(tmp_file.buf, filename.buf);
+	return finalize_object_file_flags(tmp_file.buf, filename.buf,
+					  FOF_SKIP_COLLISION_CHECK);
 }
 
 static int freshen_loose_object(const struct object_id *oid)
@@ -2350,7 +2410,8 @@ int stream_loose_object(struct input_stream *in_stream, size_t len,
 		strbuf_release(&dir);
 	}
 
-	err = finalize_object_file(tmp_file.buf, filename.buf);
+	err = finalize_object_file_flags(tmp_file.buf, filename.buf,
+					 FOF_SKIP_COLLISION_CHECK);
 	if (!err && compat)
 		err = repo_add_loose_object_map(the_repository, oid, &compat_oid);
 cleanup:
diff --git a/object-file.h b/object-file.h
index d6414610f8..81b30d269c 100644
--- a/object-file.h
+++ b/object-file.h
@@ -117,7 +117,13 @@ int check_object_signature(struct repository *r, const struct object_id *oid,
  */
 int stream_object_signature(struct repository *r, const struct object_id *oid);
 
+enum finalize_object_file_flags {
+	FOF_SKIP_COLLISION_CHECK = 1,
+};
+
 int finalize_object_file(const char *tmpfile, const char *filename);
+int finalize_object_file_flags(const char *tmpfile, const char *filename,
+			       enum finalize_object_file_flags flags);
 
 /* Helper to check and "touch" a file */
 int check_and_freshen_file(const char *fn, int freshen);
diff --git a/tmp-objdir.c b/tmp-objdir.c
index c2fb9f9193..9da0071cba 100644
--- a/tmp-objdir.c
+++ b/tmp-objdir.c
@@ -206,9 +206,11 @@ static int read_dir_paths(struct string_list *out, const char *path)
 	return 0;
 }
 
-static int migrate_paths(struct strbuf *src, struct strbuf *dst);
+static int migrate_paths(struct strbuf *src, struct strbuf *dst,
+			 enum finalize_object_file_flags flags);
 
-static int migrate_one(struct strbuf *src, struct strbuf *dst)
+static int migrate_one(struct strbuf *src, struct strbuf *dst,
+		       enum finalize_object_file_flags flags)
 {
 	struct stat st;
 
@@ -220,12 +222,18 @@ static int migrate_one(struct strbuf *src, struct strbuf *dst)
 				return -1;
 		} else if (errno != EEXIST)
 			return -1;
-		return migrate_paths(src, dst);
+		return migrate_paths(src, dst, flags);
 	}
-	return finalize_object_file(src->buf, dst->buf);
+	return finalize_object_file_flags(src->buf, dst->buf, flags);
 }
 
-static int migrate_paths(struct strbuf *src, struct strbuf *dst)
+static int is_loose_object_shard(const char *name)
+{
+	return strlen(name) == 2 && isxdigit(name[0]) && isxdigit(name[1]);
+}
+
+static int migrate_paths(struct strbuf *src, struct strbuf *dst,
+			 enum finalize_object_file_flags flags)
 {
 	size_t src_len = src->len, dst_len = dst->len;
 	struct string_list paths = STRING_LIST_INIT_DUP;
@@ -239,11 +247,15 @@ static int migrate_paths(struct strbuf *src, struct strbuf *dst)
 
 	for (i = 0; i < paths.nr; i++) {
 		const char *name = paths.items[i].string;
+		enum finalize_object_file_flags flags_copy = flags;
 
 		strbuf_addf(src, "/%s", name);
 		strbuf_addf(dst, "/%s", name);
 
-		ret |= migrate_one(src, dst);
+		if (is_loose_object_shard(name))
+			flags_copy |= FOF_SKIP_COLLISION_CHECK;
+
+		ret |= migrate_one(src, dst, flags_copy);
 
 		strbuf_setlen(src, src_len);
 		strbuf_setlen(dst, dst_len);
@@ -271,7 +283,7 @@ int tmp_objdir_migrate(struct tmp_objdir *t)
 	strbuf_addbuf(&src, &t->path);
 	strbuf_addstr(&dst, repo_get_object_directory(the_repository));
 
-	ret = migrate_paths(&src, &dst);
+	ret = migrate_paths(&src, &dst, 0);
 
 	strbuf_release(&src);
 	strbuf_release(&dst);
-- 
2.46.1.507.gffd0c9a15b2.dirty

[PATCH v5 4/8] pack-objects: use finalize_object_file() to rename pack/idx/etc

[Taylor Blau]

In most places that write files to the object database (even packfiles
via index-pack or fast-import), we use finalize_object_file(). This
prefers link()/unlink() over rename(), because it means we will prefer
data that is already in the repository to data that we are newly
writing.

We should do the same thing in pack-objects. Even though we don't think
of it as accepting outside data (and thus not being susceptible to
collision attacks), in theory a determined attacker could present just
the right set of objects to cause an incremental repack to generate
a pack with their desired hash.

This has some test and real-world fallout, as seen in the adjustment to
t5303 below. That test script assumes that we can "fix" corruption by
repacking into a good state, including when the pack generated by that
repack operation collides with a (corrupted) pack with the same hash.
This violates our assumption from the previous adjustments to
finalize_object_file() that if we're moving a new file over an existing
one, that since their checksums match, so too must their contents.

This makes "fixing" corruption like this a more explicit operation,
since the test (and users, who may fix real-life corruption using a
similar technique) must first move the broken contents out of the way.

Note also that we now call adjust_shared_perm() twice. We already call
adjust_shared_perm() in stage_tmp_packfiles(), and now call it again in
finalize_object_file(). This is somewhat wasteful, but cleaning up the
existing calls to adjust_shared_perm() is tricky (because sometimes
we're writing to a tmpfile, and sometimes we're writing directly into
the final destination), so let's tolerate some minor waste until we can
more carefully clean up the now-redundant calls.

Co-authored-by: Jeff King <peff@peff.net>
Signed-off-by: Jeff King <peff@peff.net>
Signed-off-by: Taylor Blau <me@ttaylorr.com>
---
 pack-write.c                          | 7 ++++---
 t/t5303-pack-corruption-resilience.sh | 7 ++++++-
 2 files changed, 10 insertions(+), 4 deletions(-)

diff --git a/pack-write.c b/pack-write.c
index 27965672f1..f415604c15 100644
--- a/pack-write.c
+++ b/pack-write.c
@@ -8,6 +8,7 @@
 #include "csum-file.h"
 #include "remote.h"
 #include "chunk-format.h"
+#include "object-file.h"
 #include "pack-mtimes.h"
 #include "pack-objects.h"
 #include "pack-revindex.h"
@@ -528,9 +529,9 @@ static void rename_tmp_packfile(struct strbuf *name_prefix, const char *source,
 	size_t name_prefix_len = name_prefix->len;
 
 	strbuf_addstr(name_prefix, ext);
-	if (rename(source, name_prefix->buf))
-		die_errno("unable to rename temporary file to '%s'",
-			  name_prefix->buf);
+	if (finalize_object_file(source, name_prefix->buf))
+		die("unable to rename temporary file to '%s'",
+		    name_prefix->buf);
 	strbuf_setlen(name_prefix, name_prefix_len);
 }
 
diff --git a/t/t5303-pack-corruption-resilience.sh b/t/t5303-pack-corruption-resilience.sh
index 61469ef4a6..e6a43ec9ae 100755
--- a/t/t5303-pack-corruption-resilience.sh
+++ b/t/t5303-pack-corruption-resilience.sh
@@ -44,9 +44,14 @@ create_new_pack() {
 }
 
 do_repack() {
+    for f in $pack.*
+    do
+	    mv $f "$(echo $f | sed -e 's/pack-/pack-corrupt-/')" || return 1
+    done &&
     pack=$(printf "$blob_1\n$blob_2\n$blob_3\n" |
           git pack-objects $@ .git/objects/pack/pack) &&
-    pack=".git/objects/pack/pack-${pack}"
+    pack=".git/objects/pack/pack-${pack}" &&
+    rm -f .git/objects/pack/pack-corrupt-*
 }
 
 do_corrupt_object() {
-- 
2.46.1.507.gffd0c9a15b2.dirty

[PATCH v5 5/8] sha1: do not redefine `platform_SHA_CTX` and friends

[Taylor Blau]

Our in-tree SHA-1 wrappers all define platform_SHA_CTX and related
macros to point at the opaque "context" type, init, update, and similar
functions for each specific implementation.

In hash.h, we use these platform_ variables to set up the function
pointers for, e.g., the_hash_algo->init_fn(), etc.

But while these header files have a header-specific macro that prevents
them declaring their structs / functions multiple times, they
unconditionally define the platform variables, making it impossible to
load multiple SHA-1 implementations at once.

As a prerequisite for loading a separate SHA-1 implementation for
non-cryptographic uses, only define the platform_ variables if they have
not already been defined.

Signed-off-by: Taylor Blau <me@ttaylorr.com>
---
 block-sha1/sha1.h | 2 ++
 sha1/openssl.h    | 2 ++
 sha1dc_git.h      | 3 +++
 3 files changed, 7 insertions(+)

diff --git a/block-sha1/sha1.h b/block-sha1/sha1.h
index 9fb0441b98..47bb916636 100644
--- a/block-sha1/sha1.h
+++ b/block-sha1/sha1.h
@@ -16,7 +16,9 @@ void blk_SHA1_Init(blk_SHA_CTX *ctx);
 void blk_SHA1_Update(blk_SHA_CTX *ctx, const void *dataIn, size_t len);
 void blk_SHA1_Final(unsigned char hashout[20], blk_SHA_CTX *ctx);
 
+#ifndef platform_SHA_CTX
 #define platform_SHA_CTX	blk_SHA_CTX
 #define platform_SHA1_Init	blk_SHA1_Init
 #define platform_SHA1_Update	blk_SHA1_Update
 #define platform_SHA1_Final	blk_SHA1_Final
+#endif
diff --git a/sha1/openssl.h b/sha1/openssl.h
index 006c1f4ba5..1038af47da 100644
--- a/sha1/openssl.h
+++ b/sha1/openssl.h
@@ -40,10 +40,12 @@ static inline void openssl_SHA1_Clone(struct openssl_SHA1_CTX *dst,
 	EVP_MD_CTX_copy_ex(dst->ectx, src->ectx);
 }
 
+#ifndef platform_SHA_CTX
 #define platform_SHA_CTX openssl_SHA1_CTX
 #define platform_SHA1_Init openssl_SHA1_Init
 #define platform_SHA1_Clone openssl_SHA1_Clone
 #define platform_SHA1_Update openssl_SHA1_Update
 #define platform_SHA1_Final openssl_SHA1_Final
+#endif
 
 #endif /* SHA1_OPENSSL_H */
diff --git a/sha1dc_git.h b/sha1dc_git.h
index 60e3ce8439..f6f880cabe 100644
--- a/sha1dc_git.h
+++ b/sha1dc_git.h
@@ -18,7 +18,10 @@ void git_SHA1DCFinal(unsigned char [20], SHA1_CTX *);
 void git_SHA1DCUpdate(SHA1_CTX *ctx, const void *data, unsigned long len);
 
 #define platform_SHA_IS_SHA1DC /* used by "test-tool sha1-is-sha1dc" */
+
+#ifndef platform_SHA_CTX
 #define platform_SHA_CTX SHA1_CTX
 #define platform_SHA1_Init git_SHA1DCInit
 #define platform_SHA1_Update git_SHA1DCUpdate
 #define platform_SHA1_Final git_SHA1DCFinal
+#endif
-- 
2.46.1.507.gffd0c9a15b2.dirty

[PATCH v5 6/8] hash.h: scaffolding for _unsafe hashing variants

[Taylor Blau]

Git's default SHA-1 implementation is collision-detecting, which hardens
us against known SHA-1 attacks against Git objects. This makes Git
object writes safer at the expense of some speed when hashing through
the collision-detecting implementation, which is slower than
non-collision detecting alternatives.

Prepare for loading a separate "unsafe" SHA-1 implementation that can be
used for non-cryptographic purposes, like computing the checksum of
files that use the hashwrite() API.

This commit does not actually introduce any new compile-time knobs to
control which implementation is used as the unsafe SHA-1 variant, but
does add scaffolding so that the "git_hash_algo" structure has five new
function pointers which are "unsafe" variants of the five existing
hashing-related function pointers:

  - git_hash_init_fn unsafe_init_fn
  - git_hash_clone_fn unsafe_clone_fn
  - git_hash_update_fn unsafe_update_fn
  - git_hash_final_fn unsafe_final_fn
  - git_hash_final_oid_fn unsafe_final_oid_fn

The following commit will introduce compile-time knobs to specify which
SHA-1 implementation is used for non-cryptographic uses.

Signed-off-by: Taylor Blau <me@ttaylorr.com>
---
 hash.h        | 42 ++++++++++++++++++++++++++++++++++++++++++
 object-file.c | 42 ++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 84 insertions(+)

diff --git a/hash.h b/hash.h
index 72ffbc862e..96458b129f 100644
--- a/hash.h
+++ b/hash.h
@@ -44,14 +44,32 @@
 #define platform_SHA1_Final    	SHA1_Final
 #endif
 
+#ifndef platform_SHA_CTX_unsafe
+#  define platform_SHA_CTX_unsafe      platform_SHA_CTX
+#  define platform_SHA1_Init_unsafe    platform_SHA1_Init
+#  define platform_SHA1_Update_unsafe  platform_SHA1_Update
+#  define platform_SHA1_Final_unsafe   platform_SHA1_Final
+#  ifdef platform_SHA1_Clone
+#    define platform_SHA1_Clone_unsafe platform_SHA1_Clone
+#  endif
+#endif
+
 #define git_SHA_CTX		platform_SHA_CTX
 #define git_SHA1_Init		platform_SHA1_Init
 #define git_SHA1_Update		platform_SHA1_Update
 #define git_SHA1_Final		platform_SHA1_Final
 
+#define git_SHA_CTX_unsafe	platform_SHA_CTX_unsafe
+#define git_SHA1_Init_unsafe	platform_SHA1_Init_unsafe
+#define git_SHA1_Update_unsafe	platform_SHA1_Update_unsafe
+#define git_SHA1_Final_unsafe	platform_SHA1_Final_unsafe
+
 #ifdef platform_SHA1_Clone
 #define git_SHA1_Clone	platform_SHA1_Clone
 #endif
+#ifdef platform_SHA1_Clone_unsafe
+#  define git_SHA1_Clone_unsafe platform_SHA1_Clone_unsafe
+#endif
 
 #ifndef platform_SHA256_CTX
 #define platform_SHA256_CTX	SHA256_CTX
@@ -81,6 +99,13 @@ static inline void git_SHA1_Clone(git_SHA_CTX *dst, const git_SHA_CTX *src)
 	memcpy(dst, src, sizeof(*dst));
 }
 #endif
+#ifndef SHA1_NEEDS_CLONE_HELPER_UNSAFE
+static inline void git_SHA1_Clone_unsafe(git_SHA_CTX_unsafe *dst,
+				       const git_SHA_CTX_unsafe *src)
+{
+	memcpy(dst, src, sizeof(*dst));
+}
+#endif
 
 #ifndef SHA256_NEEDS_CLONE_HELPER
 static inline void git_SHA256_Clone(git_SHA256_CTX *dst, const git_SHA256_CTX *src)
@@ -178,6 +203,8 @@ enum get_oid_result {
 /* A suitably aligned type for stack allocations of hash contexts. */
 union git_hash_ctx {
 	git_SHA_CTX sha1;
+	git_SHA_CTX_unsafe sha1_unsafe;
+
 	git_SHA256_CTX sha256;
 };
 typedef union git_hash_ctx git_hash_ctx;
@@ -222,6 +249,21 @@ struct git_hash_algo {
 	/* The hash finalization function for object IDs. */
 	git_hash_final_oid_fn final_oid_fn;
 
+	/* The non-cryptographic hash initialization function. */
+	git_hash_init_fn unsafe_init_fn;
+
+	/* The non-cryptographic hash context cloning function. */
+	git_hash_clone_fn unsafe_clone_fn;
+
+	/* The non-cryptographic hash update function. */
+	git_hash_update_fn unsafe_update_fn;
+
+	/* The non-cryptographic hash finalization function. */
+	git_hash_final_fn unsafe_final_fn;
+
+	/* The non-cryptographic hash finalization function. */
+	git_hash_final_oid_fn unsafe_final_oid_fn;
+
 	/* The OID of the empty tree. */
 	const struct object_id *empty_tree;
 
diff --git a/object-file.c b/object-file.c
index b9a3a1f62d..196c9e2df8 100644
--- a/object-file.c
+++ b/object-file.c
@@ -115,6 +115,33 @@ static void git_hash_sha1_final_oid(struct object_id *oid, git_hash_ctx *ctx)
 	oid->algo = GIT_HASH_SHA1;
 }
 
+static void git_hash_sha1_init_unsafe(git_hash_ctx *ctx)
+{
+	git_SHA1_Init_unsafe(&ctx->sha1_unsafe);
+}
+
+static void git_hash_sha1_clone_unsafe(git_hash_ctx *dst, const git_hash_ctx *src)
+{
+	git_SHA1_Clone_unsafe(&dst->sha1_unsafe, &src->sha1_unsafe);
+}
+
+static void git_hash_sha1_update_unsafe(git_hash_ctx *ctx, const void *data,
+				      size_t len)
+{
+	git_SHA1_Update_unsafe(&ctx->sha1_unsafe, data, len);
+}
+
+static void git_hash_sha1_final_unsafe(unsigned char *hash, git_hash_ctx *ctx)
+{
+	git_SHA1_Final_unsafe(hash, &ctx->sha1_unsafe);
+}
+
+static void git_hash_sha1_final_oid_unsafe(struct object_id *oid, git_hash_ctx *ctx)
+{
+	git_SHA1_Final_unsafe(oid->hash, &ctx->sha1_unsafe);
+	memset(oid->hash + GIT_SHA1_RAWSZ, 0, GIT_MAX_RAWSZ - GIT_SHA1_RAWSZ);
+	oid->algo = GIT_HASH_SHA1;
+}
 
 static void git_hash_sha256_init(git_hash_ctx *ctx)
 {
@@ -189,6 +216,11 @@ const struct git_hash_algo hash_algos[GIT_HASH_NALGOS] = {
 		.update_fn = git_hash_unknown_update,
 		.final_fn = git_hash_unknown_final,
 		.final_oid_fn = git_hash_unknown_final_oid,
+		.unsafe_init_fn = git_hash_unknown_init,
+		.unsafe_clone_fn = git_hash_unknown_clone,
+		.unsafe_update_fn = git_hash_unknown_update,
+		.unsafe_final_fn = git_hash_unknown_final,
+		.unsafe_final_oid_fn = git_hash_unknown_final_oid,
 		.empty_tree = NULL,
 		.empty_blob = NULL,
 		.null_oid = NULL,
@@ -204,6 +236,11 @@ const struct git_hash_algo hash_algos[GIT_HASH_NALGOS] = {
 		.update_fn = git_hash_sha1_update,
 		.final_fn = git_hash_sha1_final,
 		.final_oid_fn = git_hash_sha1_final_oid,
+		.unsafe_init_fn = git_hash_sha1_init_unsafe,
+		.unsafe_clone_fn = git_hash_sha1_clone_unsafe,
+		.unsafe_update_fn = git_hash_sha1_update_unsafe,
+		.unsafe_final_fn = git_hash_sha1_final_unsafe,
+		.unsafe_final_oid_fn = git_hash_sha1_final_oid_unsafe,
 		.empty_tree = &empty_tree_oid,
 		.empty_blob = &empty_blob_oid,
 		.null_oid = &null_oid_sha1,
@@ -219,6 +256,11 @@ const struct git_hash_algo hash_algos[GIT_HASH_NALGOS] = {
 		.update_fn = git_hash_sha256_update,
 		.final_fn = git_hash_sha256_final,
 		.final_oid_fn = git_hash_sha256_final_oid,
+		.unsafe_init_fn = git_hash_sha256_init,
+		.unsafe_clone_fn = git_hash_sha256_clone,
+		.unsafe_update_fn = git_hash_sha256_update,
+		.unsafe_final_fn = git_hash_sha256_final,
+		.unsafe_final_oid_fn = git_hash_sha256_final_oid,
 		.empty_tree = &empty_tree_oid_sha256,
 		.empty_blob = &empty_blob_oid_sha256,
 		.null_oid = &null_oid_sha256,
-- 
2.46.1.507.gffd0c9a15b2.dirty

[PATCH v5 7/8] Makefile: allow specifying a SHA-1 for non-cryptographic uses

[Taylor Blau]

Introduce _UNSAFE variants of the OPENSSL_SHA1, BLK_SHA1, and
APPLE_COMMON_CRYPTO_SHA1 compile-time knobs which indicate which SHA-1
implementation is to be used for non-cryptographic uses.

There are a couple of small implementation notes worth mentioning:

  - There is no way to select the collision detecting SHA-1 as the
    "fast" fallback, since the fast fallback is only for
    non-cryptographic uses, and is meant to be faster than our
    collision-detecting implementation.

  - There are no similar knobs for SHA-256, since no collision attacks
    are presently known and thus no collision-detecting implementations
    actually exist.

Signed-off-by: Taylor Blau <me@ttaylorr.com>
---
 Makefile | 25 +++++++++++++++++++++++++
 hash.h   | 30 ++++++++++++++++++++++++++++++
 2 files changed, 55 insertions(+)

diff --git a/Makefile b/Makefile
index 9cf2be070f..fb84a87592 100644
--- a/Makefile
+++ b/Makefile
@@ -521,6 +521,10 @@ include shared.mak
 # Define APPLE_COMMON_CRYPTO_SHA1 to use Apple's CommonCrypto for
 # SHA-1.
 #
+# Define the same Makefile knobs as above, but suffixed with _UNSAFE to
+# use the corresponding implementations for unsafe SHA-1 hashing for
+# non-cryptographic purposes.
+#
 # If don't enable any of the *_SHA1 settings in this section, Git will
 # default to its built-in sha1collisiondetection library, which is a
 # collision-detecting sha1 This is slower, but may detect attempted
@@ -1996,6 +2000,27 @@ endif
 endif
 endif
 
+ifdef OPENSSL_SHA1_UNSAFE
+ifndef OPENSSL_SHA1
+	EXTLIBS += $(LIB_4_CRYPTO)
+	BASIC_CFLAGS += -DSHA1_OPENSSL_UNSAFE
+endif
+else
+ifdef BLK_SHA1_UNSAFE
+ifndef BLK_SHA1
+	LIB_OBJS += block-sha1/sha1.o
+	BASIC_CFLAGS += -DSHA1_BLK_UNSAFE
+endif
+else
+ifdef APPLE_COMMON_CRYPTO_SHA1_UNSAFE
+ifndef APPLE_COMMON_CRYPTO_SHA1
+	COMPAT_CFLAGS += -DCOMMON_DIGEST_FOR_OPENSSL
+	BASIC_CFLAGS += -DSHA1_APPLE_UNSAFE
+endif
+endif
+endif
+endif
+
 ifdef OPENSSL_SHA256
 	EXTLIBS += $(LIB_4_CRYPTO)
 	BASIC_CFLAGS += -DSHA256_OPENSSL
diff --git a/hash.h b/hash.h
index 96458b129f..f97f858307 100644
--- a/hash.h
+++ b/hash.h
@@ -15,6 +15,36 @@
 #include "block-sha1/sha1.h"
 #endif
 
+#if defined(SHA1_APPLE_UNSAFE)
+#  include <CommonCrypto/CommonDigest.h>
+#  define platform_SHA_CTX_unsafe CC_SHA1_CTX
+#  define platform_SHA1_Init_unsafe CC_SHA1_Init
+#  define platform_SHA1_Update_unsafe CC_SHA1_Update
+#  define platform_SHA1_Final_unsafe CC_SHA1_Final
+#elif defined(SHA1_OPENSSL_UNSAFE)
+#  include <openssl/sha.h>
+#  if defined(OPENSSL_API_LEVEL) && OPENSSL_API_LEVEL >= 3
+#    define SHA1_NEEDS_CLONE_HELPER_UNSAFE
+#    include "sha1/openssl.h"
+#    define platform_SHA_CTX_unsafe openssl_SHA1_CTX
+#    define platform_SHA1_Init_unsafe openssl_SHA1_Init
+#    define platform_SHA1_Clone_unsafe openssl_SHA1_Clone
+#    define platform_SHA1_Update_unsafe openssl_SHA1_Update
+#    define platform_SHA1_Final_unsafe openssl_SHA1_Final
+#  else
+#    define platform_SHA_CTX_unsafe SHA_CTX
+#    define platform_SHA1_Init_unsafe SHA1_Init
+#    define platform_SHA1_Update_unsafe SHA1_Update
+#    define platform_SHA1_Final_unsafe SHA1_Final
+#  endif
+#elif defined(SHA1_BLK_UNSAFE)
+#  include "block-sha1/sha1.h"
+#  define platform_SHA_CTX_unsafe blk_SHA_CTX
+#  define platform_SHA1_Init_unsafe blk_SHA1_Init
+#  define platform_SHA1_Update_unsafe blk_SHA1_Update
+#  define platform_SHA1_Final_unsafe blk_SHA1_Final
+#endif
+
 #if defined(SHA256_NETTLE)
 #include "sha256/nettle.h"
 #elif defined(SHA256_GCRYPT)
-- 
2.46.1.507.gffd0c9a15b2.dirty

[PATCH v5 8/8] csum-file.c: use unsafe SHA-1 implementation when available

[Taylor Blau]

Update hashwrite() and friends to use the unsafe_-variants of hashing
functions, calling for e.g., "the_hash_algo->unsafe_update_fn()" instead
of "the_hash_algo->update_fn()".

These callers only use the_hash_algo to produce a checksum, which we
depend on for data integrity, but not for cryptographic purposes, so
these callers are safe to use the unsafe (non-collision detecting) SHA-1
implementation.

To time this, I took a freshly packed copy of linux.git, and ran the
following with and without the OPENSSL_SHA1_UNSAFE=1 build-knob. Both
versions were compiled with -O3:

    $ git for-each-ref --format='%(objectname)' refs/heads refs/tags >in
    $ valgrind --tool=callgrind ~/src/git/git-pack-objects \
        --revs --stdout --all-progress --use-bitmap-index <in >/dev/null

Without OPENSSL_SHA1_UNSAFE=1 (that is, using the collision-detecting
SHA-1 implementation for both cryptographic and non-cryptographic
purposes), we spend a significant amount of our instruction count in
hashwrite():

    $ callgrind_annotate --inclusive=yes | grep hashwrite | head -n1
    159,998,868,413 (79.42%)  /home/ttaylorr/src/git/csum-file.c:hashwrite [/home/ttaylorr/src/git/git-pack-objects]

, and the resulting "clone" takes 19.219 seconds of wall clock time,
18.94 seconds of user time and 0.28 seconds of system time.

Compiling with OPENSSL_SHA1_UNSAFE=1, we spend ~60% fewer instructions
in hashwrite():

    $ callgrind_annotate --inclusive=yes | grep hashwrite | head -n1
     59,164,001,176 (58.79%)  /home/ttaylorr/src/git/csum-file.c:hashwrite [/home/ttaylorr/src/git/git-pack-objects]

, and generate the resulting "clone" much faster, in only 11.597 seconds
of wall time, 11.37 seconds of user time, and 0.23 seconds of system
time, for a ~40% speed-up.

Signed-off-by: Taylor Blau <me@ttaylorr.com>
---
 csum-file.c | 18 +++++++++---------
 1 file changed, 9 insertions(+), 9 deletions(-)

diff --git a/csum-file.c b/csum-file.c
index bf82ad8f9f..c203ebf11b 100644
--- a/csum-file.c
+++ b/csum-file.c
@@ -50,7 +50,7 @@ void hashflush(struct hashfile *f)
 
 	if (offset) {
 		if (!f->skip_hash)
-			the_hash_algo->update_fn(&f->ctx, f->buffer, offset);
+			the_hash_algo->unsafe_update_fn(&f->ctx, f->buffer, offset);
 		flush(f, f->buffer, offset);
 		f->offset = 0;
 	}
@@ -73,7 +73,7 @@ int finalize_hashfile(struct hashfile *f, unsigned char *result,
 	if (f->skip_hash)
 		hashclr(f->buffer, the_repository->hash_algo);
 	else
-		the_hash_algo->final_fn(f->buffer, &f->ctx);
+		the_hash_algo->unsafe_final_fn(f->buffer, &f->ctx);
 
 	if (result)
 		hashcpy(result, f->buffer, the_repository->hash_algo);
@@ -128,7 +128,7 @@ void hashwrite(struct hashfile *f, const void *buf, unsigned int count)
 			 * f->offset is necessarily zero.
 			 */
 			if (!f->skip_hash)
-				the_hash_algo->update_fn(&f->ctx, buf, nr);
+				the_hash_algo->unsafe_update_fn(&f->ctx, buf, nr);
 			flush(f, buf, nr);
 		} else {
 			/*
@@ -174,7 +174,7 @@ static struct hashfile *hashfd_internal(int fd, const char *name,
 	f->name = name;
 	f->do_crc = 0;
 	f->skip_hash = 0;
-	the_hash_algo->init_fn(&f->ctx);
+	the_hash_algo->unsafe_init_fn(&f->ctx);
 
 	f->buffer_len = buffer_len;
 	f->buffer = xmalloc(buffer_len);
@@ -208,7 +208,7 @@ void hashfile_checkpoint(struct hashfile *f, struct hashfile_checkpoint *checkpo
 {
 	hashflush(f);
 	checkpoint->offset = f->total;
-	the_hash_algo->clone_fn(&checkpoint->ctx, &f->ctx);
+	the_hash_algo->unsafe_clone_fn(&checkpoint->ctx, &f->ctx);
 }
 
 int hashfile_truncate(struct hashfile *f, struct hashfile_checkpoint *checkpoint)
@@ -219,7 +219,7 @@ int hashfile_truncate(struct hashfile *f, struct hashfile_checkpoint *checkpoint
 	    lseek(f->fd, offset, SEEK_SET) != offset)
 		return -1;
 	f->total = offset;
-	the_hash_algo->clone_fn(&f->ctx, &checkpoint->ctx);
+	the_hash_algo->unsafe_clone_fn(&f->ctx, &checkpoint->ctx);
 	f->offset = 0; /* hashflush() was called in checkpoint */
 	return 0;
 }
@@ -245,9 +245,9 @@ int hashfile_checksum_valid(const unsigned char *data, size_t total_len)
 	if (total_len < the_hash_algo->rawsz)
 		return 0; /* say "too short"? */
 
-	the_hash_algo->init_fn(&ctx);
-	the_hash_algo->update_fn(&ctx, data, data_len);
-	the_hash_algo->final_fn(got, &ctx);
+	the_hash_algo->unsafe_init_fn(&ctx);
+	the_hash_algo->unsafe_update_fn(&ctx, data, data_len);
+	the_hash_algo->unsafe_final_fn(got, &ctx);
 
 	return hasheq(got, data + data_len, the_repository->hash_algo);
 }
-- 
2.46.1.507.gffd0c9a15b2.dirty

Re: [PATCH v5 0/8] hash.h: support choosing a separate SHA-1 for non-cryptographic uses

[Elijah Newren]

Hi...

On Thu, Sep 26, 2024 at 8:22 AM Taylor Blau <me@ttaylorr.com> wrote:
...
> I think most of the review dust has settled up to this point, so I'm
> imagining that this is the final version of this series for now, or at
> least very close to it. But if something new comes up, please let me
> know!
>
...
> Range-diff against v4:
> -:  ---------- > 1:  6f1ee91fff finalize_object_file(): check for name collision before renaming
> -:  ---------- > 2:  133047ca8c finalize_object_file(): refactor unlink_or_warn() placement
> 1:  ed9eeef851 ! 3:  41d38352a4 finalize_object_file(): implement collision check
>     @@ Commit message
>              object name, so checking for collisions at the content level doesn't
>              add anything.
>
>     -        This is why we do not bother to check the inflated object contents
>     -        for collisions either, since either (a) the object contents have the
>     -        fingerprint of a SHA-1 collision, in which case the collision
>     -        detecting SHA-1 implementation used to hash the contents to give us
>     -        a path would have already rejected it, or (b) the contents are part
>     -        of a colliding pair which does not bear the same fingerprints of
>     -        known collision attacks, in which case we would not have caught it
>     -        anyway.
>     +        Adding a content-level collision check would have to happen at a
>     +        higher level than in finalize_object_file(), since (avoiding race
>     +        conditions) writing an object loose which already exists in the
>     +        repository will prevent us from even reaching finalize_object_file()
>     +        via the object freshening code.
>     +
>     +        There is a collision check in index-pack via its `check_collision()`
>     +        function, but there isn't an analogous function in unpack-objects,
>     +        which just feeds the result to write_object_file().
>
>              So skipping the collision check here does not change for better or
>              worse the hardness of loose object writes.
>     @@ Commit message
>
>          Co-authored-by: Jeff King <peff@peff.net>
>          Signed-off-by: Jeff King <peff@peff.net>
>     +    Helped-by: Elijah Newren <newren@gmail.com>
>          Signed-off-by: Taylor Blau <me@ttaylorr.com>
>
>       ## object-file.c ##
> 2:  3cc7f7b1f6 = 4:  611475d83e pack-objects: use finalize_object_file() to rename pack/idx/etc
> 3:  8f8ac0f5b0 = 5:  9913a5d971 sha1: do not redefine `platform_SHA_CTX` and friends
> 4:  d300e9c688 = 6:  65de6d724d hash.h: scaffolding for _unsafe hashing variants
> 5:  af8fd9aa4e = 7:  3884cd0e3a Makefile: allow specifying a SHA-1 for non-cryptographic uses
> 6:  4b83dd05e9 ! 8:  62abddf73d csum-file.c: use unsafe SHA-1 implementation when available
>     @@ Commit message
>
>          These callers only use the_hash_algo to produce a checksum, which we
>          depend on for data integrity, but not for cryptographic purposes, so
>     -    these callers are safe to use the unsafe (and potentially non-collision
>     -    detecting) SHA-1 implementation.
>     +    these callers are safe to use the unsafe (non-collision detecting) SHA-1
>     +    implementation.
>
>          To time this, I took a freshly packed copy of linux.git, and ran the
>          following with and without the OPENSSL_SHA1_UNSAFE=1 build-knob. Both
>     @@ Commit message
>              $ callgrind_annotate --inclusive=yes | grep hashwrite | head -n1
>               59,164,001,176 (58.79%)  /home/ttaylorr/src/git/csum-file.c:hashwrite [/home/ttaylorr/src/git/git-pack-objects]
>
>     -    , and generate the resulting "clone" much unsafeer, in only 11.597 seconds
>     +    , and generate the resulting "clone" much faster, in only 11.597 seconds
>          of wall time, 11.37 seconds of user time, and 0.23 seconds of system
>          time, for a ~40% speed-up.

This round looks good to me.

Re: [PATCH v5 0/8] hash.h: support choosing a separate SHA-1 for non-cryptographic uses

[Junio C Hamano]

Elijah Newren <newren@gmail.com> writes:

> Hi...
>
>> ...
>>     -    , and generate the resulting "clone" much unsafeer, in only 11.597 seconds
>>     +    , and generate the resulting "clone" much faster, in only 11.597 seconds
>>          of wall time, 11.37 seconds of user time, and 0.23 seconds of system
>>          time, for a ~40% speed-up.
>
> This round looks good to me.

Thanks, both.

Re: [PATCH v5 0/8] hash.h: support choosing a separate SHA-1 for non-cryptographic uses

[Jeff King]

On Thu, Sep 26, 2024 at 11:22:27AM -0400, Taylor Blau wrote:

> Range-diff against v4:
> -:  ---------- > 1:  6f1ee91fff finalize_object_file(): check for name collision before renaming
> -:  ---------- > 2:  133047ca8c finalize_object_file(): refactor unlink_or_warn() placement
> 1:  ed9eeef851 ! 3:  41d38352a4 finalize_object_file(): implement collision check
> [...]
> 2:  3cc7f7b1f6 = 4:  611475d83e pack-objects: use finalize_object_file() to rename pack/idx/etc
> 3:  8f8ac0f5b0 = 5:  9913a5d971 sha1: do not redefine `platform_SHA_CTX` and friends
> 4:  d300e9c688 = 6:  65de6d724d hash.h: scaffolding for _unsafe hashing variants
> 5:  af8fd9aa4e = 7:  3884cd0e3a Makefile: allow specifying a SHA-1 for non-cryptographic uses
> 6:  4b83dd05e9 ! 8:  62abddf73d csum-file.c: use unsafe SHA-1 implementation when available

Funny range-diff, but I think those patches are here in the series as
expected.

Anyway, the result looks good to me!

-Peff