Re: [PATCH 0/4] faster SHA-1 collision detection

[Sebastian Thiel <sebastian.thiel@icloud.com>]

Thanks for reeling me in, Scott!

First of all, I am very happy to see that overall, everyone here is
making an effort to find a way to speed up SHA-1dc again.
It's so impactful!


It really did hurt when I finally had to add SHA-1dc to Gitoxide and see 
the performance of clones plummet. And it still hurts me knowing that
an incredible amount of CPU time is wasted doing something that we now
know can be done much faster. At GitHub scale, this must be more than
a blip.

While it's my dream to one day have a GitHub action that uses `gix` to
clone and safe even more power, I think Git is in a far better spot
to achieve significant savings much sooner.

On 07.10.26 20:13, Scott Chacon wrote:
> Hey,
> 
> On Wed, Oct 7, 2026 at 7:23 PM Junio C Hamano <gitster@pobox.com> wrote:
>>> This series ports the approach of Sam Reis's sha1dc Rust crate [1],
>>> which gitoxide recently switched to [2], to C.
>>
>> Which means license-wise the original is compatible with us, I
>> presume, as they are "Apache2 or MIT, your choice".
>>
>> How can you/we be sure, with respect to the current AI policy in
>> SubmittingPatches (which by the way was vetted by SFC lawyers), that
>> your "AI generated" code did not "borrow" from places that gets
>> you/us into trouble?
> 
> It's a good question. I actually just submitted a proposed update to
> that policy based on SFC's updated guidelines, but either way, I
> learned about this from Sam and have talked to him about the port and
> he seemed excited about it. I can triple check, but I'm fairly
> confident that he's fine with this and I am fine signing off on it
> under the terms of the DCO language.
> 
> Of course, he in turn used AI tooling to produce _his_ library, but
> within the guidelines of the updated SFC guidelines. Johannes's
> alternative series is the original Rust code of Sam that my agent
> looked at to produce this (in addition to his blog post explaining
> it), so I'm not sure how that might be materially different.
> 
>>> The end result hashes roughly 2.7x faster on the Xeon and 2.85x faster
>>> on the M5 Max. Single-threaded index-pack of git.git goes from 24.3s to
>>> 12.7s on the Xeon, and from 16.1s to 8.7s on the M5 Max.
>>>
>>> Hashing throughput on the Xeon, in MiB/s:
>>>
>>>                                  16KiB    1MiB   vs OpenSSL
>>>    OpenSSL SHA-1 (no detection)   1234    1129      1.00x
>>>    sha1dc/ (today)                 435     450      2.67x
>>>    shani+avx2 (default here)      1002     901      1.24x
>>>    shani+sse2                     1075    1008      1.13x
>>>    portable+avx2                   553     654      1.96x
>>>    portable+sse2                   603     681      1.84x
>>>    portable                        466     565      2.29x
>>>
>>> In other words, currently collision detection costs about 1.5–2.5x on
>>> top of the hashing itself today, but only about 0.2x with the series.
>>
>> Thanks for these numbers.
> 
> It would have been better had I provided the same relative scale (it
> should be 1.5-2.5x vs 1.2x, but whatever, you probably get it. It's
> 20% overhead here vs 50%-150% overhead previously).
> 
>>> [1] https://sam.dev/blog/faster-sha1-collision-detection
>>> [2] https://github.com/GitoxideLabs/gitoxide/pull/3008
>>
>> And the pointers to the original sources.
> 
> CC'ing Sam (sha1dc rust guy) and Sebastian (Gitoxide) on this, just in
> case they have an opinion but I'm pretty sure they would be more than
> happy for this to be integrated.
> 
> Scott