SHA-accelerated Git

SHA-accelerated Git

[Jeffrey Walton]

Hi Everyone,

I have a lot of experience with cutting in SHA acceleration. I have no
experience with Git.

If someone would setup a fork and provide a call like:

    // https://github.com/torvalds/linux/blob/master/arch/x86/include/asm/cpufeature.h
    // Or whatever the project prefers
    has_x86_sha = cpu_feature_enabled(X86_FEATURE_SHA_NI);

    if (has_x86_sha)
    {
        sha1_hash_block(state, ptr, size);
    }

And provide a stub:

    sha1_hash_block(uint32_t* state, uint8_t* ptr, size_t size)
    {
    }

Then I would be happy to fill in the pieces.

I can also help with ARMv8.

In fact, if you want to use Andy Polyakov's asm from Cryptogams
(https://www.openssl.org/~appro/cryptogams/), then I can help with
that, too. Andy's skills are renowned and his code is used in the
kernel.

Jeff

Re: SHA-accelerated Git

[Michal Suchánek]

Hello,

On Sun, 2 Jun 2019 06:43:07 -0400
Jeffrey Walton <noloader@gmail.com> wrote:

> Hi Everyone,
> 
> I have a lot of experience with cutting in SHA acceleration. I have no
> experience with Git.

sha1 is dead. Git is planning to move away from it. It will not happen
overnight but it probably does not make sense to accelerate it at this
point.

https://github.com/git/git/blob/next/Documentation/technical/hash-function-transition.txt


> 
> If someone would setup a fork and provide a call like:
> 
>     // https://github.com/torvalds/linux/blob/master/arch/x86/include/asm/cpufeature.h
>     // Or whatever the project prefers
>     has_x86_sha = cpu_feature_enabled(X86_FEATURE_SHA_NI);
> 
>     if (has_x86_sha)
>     {
>         sha1_hash_block(state, ptr, size);
>     }
> 
> And provide a stub:
> 
>     sha1_hash_block(uint32_t* state, uint8_t* ptr, size_t size)
>     {
>     }
> 
> Then I would be happy to fill in the pieces.
> 
> I can also help with ARMv8.
> 
> In fact, if you want to use Andy Polyakov's asm from Cryptogams
> (https://www.openssl.org/~appro/cryptogams/), then I can help with
> that, too. Andy's skills are renowned and his code is used in the
> kernel.

And buggy. I am not referring to the sha1 implementation in particular
but to the hand-crafted assembly in general.

Do you have some performance data that shows significant improvement of
common tasks that makes maintaining this hackery worthwhile?

Thanks

Michal

Re: SHA-accelerated Git

[Jeffrey Walton]

On Sun, Jun 2, 2019 at 7:21 AM Michal Suchánek <msuchanek@suse.de> wrote:
>
> Hello,
>
> On Sun, 2 Jun 2019 06:43:07 -0400
> Jeffrey Walton <noloader@gmail.com> wrote:
>
> > Hi Everyone,
> >
> > I have a lot of experience with cutting in SHA acceleration. I have no
> > experience with Git.
>
> sha1 is dead. Git is planning to move away from it. It will not happen
> overnight but it probably does not make sense to accelerate it at this
> point.
>
> https://github.com/git/git/blob/next/Documentation/technical/hash-function-transition.txt

SHA-1 has been dead for years. That has not stopped Git yet :)

My guess is, Git will have to carry around a SHA-1 implementation for
a decade or so for interop purposes.

> > If someone would setup a fork and provide a call like:
> >
> >     // https://github.com/torvalds/linux/blob/master/arch/x86/include/asm/cpufeature.h
> >     // Or whatever the project prefers
> >     has_x86_sha = cpu_feature_enabled(X86_FEATURE_SHA_NI);
> >
> >     if (has_x86_sha)
> >     {
> >         sha1_hash_block(state, ptr, size);
> >     }
> >
> > And provide a stub:
> >
> >     sha1_hash_block(uint32_t* state, uint8_t* ptr, size_t size)
> >     {
> >     }
> >
> > Then I would be happy to fill in the pieces.
> >
> > I can also help with ARMv8.
> >
> > In fact, if you want to use Andy Polyakov's asm from Cryptogams
> > (https://www.openssl.org/~appro/cryptogams/), then I can help with
> > that, too. Andy's skills are renowned and his code is used in the
> > kernel.
>
> And buggy. I am not referring to the sha1 implementation in particular
> but to the hand-crafted assembly in general.
>
> Do you have some performance data that shows significant improvement of
> common tasks that makes maintaining this hackery worthwhile?

Well, considering Git carries along an SHA-1 implementation with
undefined behavior, anything would be an improvement since it can't
pass acceptance testing out of the box. And Git does that for a 0.01
to 0.03 cpb increase in spperformance.

For SHA-1 intrinsics on x86, performance increases to about 1.7 cpb.
For SHA-256 performance increases to 3.8 cpb. On ARMv8, performance
will increase to about 2.0 cpb on A-53 cpu's, and about 1.5 to 1.0 cpb
on A-57 cpus.

Comparing against a standard C/C++ implementation, Git will see a 3x
to 6x speedup depending on the platform and algorithm.

Jeff

Re: SHA-accelerated Git

[Johannes Schindelin]

[-- Attachment #1: Type: text/plain, Size: 1281 bytes --]

Hi Jeffrey,

On Sun, 2 Jun 2019, Jeffrey Walton wrote:

> On Sun, Jun 2, 2019 at 7:21 AM Michal Suchánek <msuchanek@suse.de> wrote:
> >
> > On Sun, 2 Jun 2019 06:43:07 -0400
> > Jeffrey Walton <noloader@gmail.com> wrote:
> >
> > > I have a lot of experience with cutting in SHA acceleration. I have
> > > no experience with Git.
> >
> > sha1 is dead. Git is planning to move away from it. It will not happen
> > overnight but it probably does not make sense to accelerate it at this
> > point.
> >
> > https://github.com/git/git/blob/next/Documentation/technical/hash-function-transition.txt
>
> SHA-1 has been dead for years. That has not stopped Git yet :)

Even worse. Git has changed from an OpenSSL accelerated SHA-1 to a
collision-detecting SHA-1 (sha1dc).

As long as your accelerated SHA-1 cannot guarantee the same level of
safeguards as sha1dc, there is a negligible likelihood that your patches
will be accepted.

> Comparing against a standard C/C++ implementation, Git will see a 3x to
> 6x speedup depending on the platform and algorithm.

Indeed, this reflects my findings.

It's just that we struck the balance on the other side of performance, in
favor of preventing possible attack vectors as best as we can.

Ciao,
Johannes