Linux cryptographic layer development
 help / color / mirror / Atom feed
From: Eric Biggers <ebiggers@kernel.org>
To: linux-crypto@vger.kernel.org
Cc: Ard Biesheuvel <ardb@kernel.org>,
	"Jason A . Donenfeld" <Jason@zx2c4.com>,
	Herbert Xu <herbert@gondor.apana.org.au>,
	x86@kernel.org, Eric Biggers <ebiggers@kernel.org>
Subject: [PATCH 6/8] lib/crypto: x86/sha512: Keep state in registers across blocks in SSSE3
Date: Sat, 21 Nov 2026 18:20:28 -0800	[thread overview]
Message-ID: <20261122022030.296468-7-ebiggers@kernel.org> (raw)
In-Reply-To: <20261122022030.296468-1-ebiggers@kernel.org>

In sha512_transform_ssse3(), each iteration of .Lupdateblock updates the
state in memory using 'add reg, mem' and then immediately reloads all 8
state words from memory at the top of the next iteration.

Hoist the initial state load before .Lupdateblock, and update the state
at the end of each block using 'add mem, reg' followed by 'mov reg, mem'
(via a new 'addm' macro), as sha512-avx2-asm.S, sha256-ssse3-asm.S, and
sha256-avx2-asm.S already do.

This keeps the updated state in registers across blocks, eliminating 8
loads per additional block and avoiding store-to-load forwarding stalls.

Signed-off-by: Eric Biggers <ebiggers@kernel.org>
---
 lib/crypto/x86/sha512-ssse3-asm.S | 26 ++++++++++++++++----------
 1 file changed, 16 insertions(+), 10 deletions(-)

diff --git a/lib/crypto/x86/sha512-ssse3-asm.S b/lib/crypto/x86/sha512-ssse3-asm.S
index f96ce3c083ea7..f7dade03c6527 100644
--- a/lib/crypto/x86/sha512-ssse3-asm.S
+++ b/lib/crypto/x86/sha512-ssse3-asm.S
@@ -100,6 +100,12 @@ XROR8 =		%xmm15	# XMM_QWORD_ROR8
 # SHA Constants (static mem)
 #define K_t(i)    8*i+K512(%rip)
 
+# Add \mem to \reg, then copy the result back to \mem.
+.macro addm mem reg
+	add	\mem, \reg
+	mov	\reg, \mem
+.endm
+
 .macro RotateXW
 	# Rotate symbols XW0..XW7 left
 	XWTMP = XW0
@@ -304,8 +310,6 @@ SYM_FUNC_START(sha512_transform_ssse3)
 	movdqa	XMM_QWORD_BSWAP(%rip), XBSWAP
 	movdqa	XMM_QWORD_ROR8(%rip), XROR8
 
-.Lupdateblock:
-
 	# Load state variables
 	mov	DIGEST(0), a_64
 	mov	DIGEST(1), b_64
@@ -316,6 +320,8 @@ SYM_FUNC_START(sha512_transform_ssse3)
 	mov	DIGEST(6), g_64
 	mov	DIGEST(7), h_64
 
+.Lupdateblock:
+
 	# Rounds 0 - 13: BSWAP W[0..15] into XW0..XW7 and compute rounds 0..13.
 	movdqu	MSG(0), XW0
 	pshufb	XBSWAP, XW0	# BSWAP
@@ -375,14 +381,14 @@ SYM_FUNC_START(sha512_transform_ssse3)
 
 	# Update digest
 	mov	0(%rsp), digest
-	add	a_64, DIGEST(0)
-	add	b_64, DIGEST(1)
-	add	c_64, DIGEST(2)
-	add	d_64, DIGEST(3)
-	add	e_64, DIGEST(4)
-	add	f_64, DIGEST(5)
-	add	g_64, DIGEST(6)
-	add	h_64, DIGEST(7)
+	addm	DIGEST(0), a_64
+	addm	DIGEST(1), b_64
+	addm	DIGEST(2), c_64
+	addm	DIGEST(3), d_64
+	addm	DIGEST(4), e_64
+	addm	DIGEST(5), f_64
+	addm	DIGEST(6), g_64
+	addm	DIGEST(7), h_64
 
 	# Decrement msglen, and advance to next message block if needed
 	add	$16*8, msg
-- 
2.56.1


  parent reply	other threads:[~2026-11-22  2:21 UTC|newest]

Thread overview: 9+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-22  2:20 [PATCH 0/8] lib/crypto: De-bloat and optimize sha512-ssse3-asm.S Eric Biggers
2026-11-22  2:20 ` [PATCH 1/8] lib/crypto: x86/sha512: Partially roll up sha512_transform_ssse3() Eric Biggers
2026-11-22  2:20 ` [PATCH 2/8] lib/crypto: x86/sha512: Keep message schedule in XMM registers Eric Biggers
2026-11-22  2:20 ` [PATCH 3/8] lib/crypto: x86/sha512: Keep W[t]+K[t] in %xmm5 instead of stack Eric Biggers
2026-11-22  2:20 ` [PATCH 4/8] lib/crypto: x86/sha512: Keep XMM_QWORD_BSWAP in %xmm14 across blocks Eric Biggers
2026-11-22  2:20 ` [PATCH 5/8] lib/crypto: x86/sha512: Use pshufb for ror64(x, 8) in SSSE3 s0 Eric Biggers
2026-11-22  2:20 ` Eric Biggers [this message]
2026-11-22  2:20 ` [PATCH 7/8] lib/crypto: x86/sha512: Shorten critical-path additions in SSSE3 Eric Biggers
2026-11-22  2:20 ` [PATCH 8/8] lib/crypto: x86/sha512: Use local labels for data Eric Biggers

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20261122022030.296468-7-ebiggers@kernel.org \
    --to=ebiggers@kernel.org \
    --cc=Jason@zx2c4.com \
    --cc=ardb@kernel.org \
    --cc=herbert@gondor.apana.org.au \
    --cc=linux-crypto@vger.kernel.org \
    --cc=x86@kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox