[PATCH] lib/crypto: x86/chacha: Remove vzeroupper at beginning of functions

[PATCH] lib/crypto: x86/chacha: Remove vzeroupper at beginning of functions

[Eric Biggers]

There's no point in executing vzeroupper at the beginning of the AVX2
and AVX-512 optimized functions.  Remove it, leaving the vzeroupper at
the end (where it is needed) only.

Signed-off-by: Eric Biggers <ebiggers@kernel.org>
---
 lib/crypto/x86/chacha-avx2-x86_64.S   | 5 -----
 lib/crypto/x86/chacha-avx512-x86_64.S | 6 ------
 2 files changed, 11 deletions(-)

diff --git a/lib/crypto/x86/chacha-avx2-x86_64.S b/lib/crypto/x86/chacha-avx2-x86_64.S
index f3d8fc018249..0b0cf92a022c 100644
--- a/lib/crypto/x86/chacha-avx2-x86_64.S
+++ b/lib/crypto/x86/chacha-avx2-x86_64.S
@@ -46,8 +46,6 @@ SYM_FUNC_START(chacha_2block_xor_avx2)
 	# on four words in each matrix in parallel, but requires shuffling to
 	# rearrange the words after each round.
 
-	vzeroupper
-
 	# x0..3[0-2] = s0..3
 	vbroadcasti128	0x00(%rdi),%ymm0
 	vbroadcasti128	0x10(%rdi),%ymm1
@@ -240,8 +238,6 @@ SYM_FUNC_START(chacha_4block_xor_avx2)
 	# required word shuffling has a rather high latency, we can do the
 	# arithmetic on two matrix-pairs without much slowdown.
 
-	vzeroupper
-
 	# x0..3[0-4] = s0..3
 	vbroadcasti128	0x00(%rdi),%ymm0
 	vbroadcasti128	0x10(%rdi),%ymm1
@@ -548,7 +544,6 @@ SYM_FUNC_START(chacha_8block_xor_avx2)
 	# rotation is done with the slightly better performing byte shuffling,
 	# 7/12-bit word rotation uses traditional shift+OR.
 
-	vzeroupper
 	# 4 * 32 byte stack, 32-byte aligned
 	lea		8(%rsp),%r10
 	and		$~31, %rsp
diff --git a/lib/crypto/x86/chacha-avx512-x86_64.S b/lib/crypto/x86/chacha-avx512-x86_64.S
index 422479631d5b..ffdfe005cba6 100644
--- a/lib/crypto/x86/chacha-avx512-x86_64.S
+++ b/lib/crypto/x86/chacha-avx512-x86_64.S
@@ -55,8 +55,6 @@ SYM_FUNC_START(chacha_2block_xor_avx512vl)
 	# on four words in each matrix in parallel, but requires shuffling to
 	# rearrange the words after each round.
 
-	vzeroupper
-
 	# x0..3[0-2] = s0..3
 	vbroadcasti128	0x00(%rdi),%ymm0
 	vbroadcasti128	0x10(%rdi),%ymm1
@@ -220,8 +218,6 @@ SYM_FUNC_START(chacha_4block_xor_avx512vl)
 	# required word shuffling has a rather high latency, we can do the
 	# arithmetic on two matrix-pairs without much slowdown.
 
-	vzeroupper
-
 	# x0..3[0-4] = s0..3
 	vbroadcasti128	0x00(%rdi),%ymm0
 	vbroadcasti128	0x10(%rdi),%ymm1
@@ -489,8 +485,6 @@ SYM_FUNC_END(chacha_4block_xor_avx512vl)
 	# VL=32 encrypts eight blocks in 256-bit ymm registers.
 	# VL=64 encrypts sixteen blocks in 512-bit zmm registers.
 
-	vzeroupper
-
 	# x0..15[0-VL/4] = s[0..15]
 	vpbroadcastd	0x00(%rdi),V0
 	vpbroadcastd	0x04(%rdi),V1

base-commit: 9fbbbaefd5f95551da18426c200cf41e1b4255a1
prerequisite-patch-id: 890b33260038bff2e3adfe7867af5919666677fc
-- 
2.56.0

Re: [PATCH] lib/crypto: x86/chacha: Remove vzeroupper at beginning of functions

[Antonio Quartulli]

Tested on Zen 5 (Ryzen 9 9950X) in a KVM guest, on top of the bzhi
patch. KUnit chacha suite + benchmark (64..16384 B, 3 runs each),
compared with the tree without this patch:

   path                 CPU setup              KUnit  perf
   -------------------  ---------------------  -----  ------------
   AVX-512, zmm         native                 pass   -0.5..+0.5%
   AVX-512VL, ymm only  spoofed SKX (6/0x55)   pass   -0.8..+0.1%
   AVX2                 clearcpuid=avx512f     pass   +0.1..+1.9%

Tested-by: Antonio Quartulli <antonio@mandelbit.com>

-- 
Antonio Quartulli

CEO and Co-Founder
Mandelbit Srl
https://www.mandelbit.com