From: Eric Biggers <ebiggers@kernel.org>
To: linux-crypto@vger.kernel.org
Cc: Ard Biesheuvel <ardb@kernel.org>,
"Jason A . Donenfeld" <Jason@zx2c4.com>,
Herbert Xu <herbert@gondor.apana.org.au>,
x86@kernel.org, Eric Biggers <ebiggers@kernel.org>
Subject: [PATCH 6/8] lib/crypto: x86/sha512: Keep state in registers across blocks in SSSE3
Date: Sat, 21 Nov 2026 18:20:28 -0800 [thread overview]
Message-ID: <20261122022030.296468-7-ebiggers@kernel.org> (raw)
In-Reply-To: <20261122022030.296468-1-ebiggers@kernel.org>
In sha512_transform_ssse3(), each iteration of .Lupdateblock updates the
state in memory using 'add reg, mem' and then immediately reloads all 8
state words from memory at the top of the next iteration.
Hoist the initial state load before .Lupdateblock, and update the state
at the end of each block using 'add mem, reg' followed by 'mov reg, mem'
(via a new 'addm' macro), as sha512-avx2-asm.S, sha256-ssse3-asm.S, and
sha256-avx2-asm.S already do.
This keeps the updated state in registers across blocks, eliminating 8
loads per additional block and avoiding store-to-load forwarding stalls.
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
---
lib/crypto/x86/sha512-ssse3-asm.S | 26 ++++++++++++++++----------
1 file changed, 16 insertions(+), 10 deletions(-)
diff --git a/lib/crypto/x86/sha512-ssse3-asm.S b/lib/crypto/x86/sha512-ssse3-asm.S
index f96ce3c083ea7..f7dade03c6527 100644
--- a/lib/crypto/x86/sha512-ssse3-asm.S
+++ b/lib/crypto/x86/sha512-ssse3-asm.S
@@ -100,6 +100,12 @@ XROR8 = %xmm15 # XMM_QWORD_ROR8
# SHA Constants (static mem)
#define K_t(i) 8*i+K512(%rip)
+# Add \mem to \reg, then copy the result back to \mem.
+.macro addm mem reg
+ add \mem, \reg
+ mov \reg, \mem
+.endm
+
.macro RotateXW
# Rotate symbols XW0..XW7 left
XWTMP = XW0
@@ -304,8 +310,6 @@ SYM_FUNC_START(sha512_transform_ssse3)
movdqa XMM_QWORD_BSWAP(%rip), XBSWAP
movdqa XMM_QWORD_ROR8(%rip), XROR8
-.Lupdateblock:
-
# Load state variables
mov DIGEST(0), a_64
mov DIGEST(1), b_64
@@ -316,6 +320,8 @@ SYM_FUNC_START(sha512_transform_ssse3)
mov DIGEST(6), g_64
mov DIGEST(7), h_64
+.Lupdateblock:
+
# Rounds 0 - 13: BSWAP W[0..15] into XW0..XW7 and compute rounds 0..13.
movdqu MSG(0), XW0
pshufb XBSWAP, XW0 # BSWAP
@@ -375,14 +381,14 @@ SYM_FUNC_START(sha512_transform_ssse3)
# Update digest
mov 0(%rsp), digest
- add a_64, DIGEST(0)
- add b_64, DIGEST(1)
- add c_64, DIGEST(2)
- add d_64, DIGEST(3)
- add e_64, DIGEST(4)
- add f_64, DIGEST(5)
- add g_64, DIGEST(6)
- add h_64, DIGEST(7)
+ addm DIGEST(0), a_64
+ addm DIGEST(1), b_64
+ addm DIGEST(2), c_64
+ addm DIGEST(3), d_64
+ addm DIGEST(4), e_64
+ addm DIGEST(5), f_64
+ addm DIGEST(6), g_64
+ addm DIGEST(7), h_64
# Decrement msglen, and advance to next message block if needed
add $16*8, msg
--
2.56.1
next prev parent reply other threads:[~2026-11-22 2:21 UTC|newest]
Thread overview: 9+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-22 2:20 [PATCH 0/8] lib/crypto: De-bloat and optimize sha512-ssse3-asm.S Eric Biggers
2026-11-22 2:20 ` [PATCH 1/8] lib/crypto: x86/sha512: Partially roll up sha512_transform_ssse3() Eric Biggers
2026-11-22 2:20 ` [PATCH 2/8] lib/crypto: x86/sha512: Keep message schedule in XMM registers Eric Biggers
2026-11-22 2:20 ` [PATCH 3/8] lib/crypto: x86/sha512: Keep W[t]+K[t] in %xmm5 instead of stack Eric Biggers
2026-11-22 2:20 ` [PATCH 4/8] lib/crypto: x86/sha512: Keep XMM_QWORD_BSWAP in %xmm14 across blocks Eric Biggers
2026-11-22 2:20 ` [PATCH 5/8] lib/crypto: x86/sha512: Use pshufb for ror64(x, 8) in SSSE3 s0 Eric Biggers
2026-11-22 2:20 ` Eric Biggers [this message]
2026-11-22 2:20 ` [PATCH 7/8] lib/crypto: x86/sha512: Shorten critical-path additions in SSSE3 Eric Biggers
2026-11-22 2:20 ` [PATCH 8/8] lib/crypto: x86/sha512: Use local labels for data Eric Biggers
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261122022030.296468-7-ebiggers@kernel.org \
--to=ebiggers@kernel.org \
--cc=Jason@zx2c4.com \
--cc=ardb@kernel.org \
--cc=herbert@gondor.apana.org.au \
--cc=linux-crypto@vger.kernel.org \
--cc=x86@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox