Re: [PATCH] lib/crypto: x86/chacha: Add a 16-block AVX-512 variant

[Eric Biggers <ebiggers@kernel.org>]

On Wed, Jul 22, 2026 at 05:32:47PM +0200, Martin Willi wrote:
> The existing AVX-512VL code processes at most eight blocks at a time
> using 256-bit ymm registers. This width was chosen deliberately to
> avoid the heavy core down-clocking that 512-bit zmm instructions
> triggered on Skylake-X.
> 
> That penalty is gone on more recent AVX-512 microarchitectures, where
> full 512-bit zmm registers can encrypt sixteen blocks per invocation
> and roughly double the data-level parallelism for bulk traffic. Add
> such a 16-block variant and dispatch to it for inputs larger than
> eight blocks, ahead of the AVX-512VL path which still handles the
> remainder.
> 
> On a Zen 5, the tcrypt speed test for chacha20 with 1024-byte blocks
> reports 7.5 GB/s with the new variant versus 4.2 GB/s for the
> AVX-512VL path, roughly a 1.8x improvement.
> 
> Enable it only on CPUs advertising AVX-512F with full zmm XSAVE state,
> and keep it disabled when X86_FEATURE_PREFER_YMM is set so
> down-clocking parts continue to use the ymm-based AVX-512VL path.
> 
> Signed-off-by: Martin Willi <martin@strongswan.org>

Thanks Martin!

This is very similar to the existing chacha_8block_xor_avx512vl(), just
with 512-bit vectors instead of 256-bit.  They even use the same CPU
instruction set extensions: there are actually no cases where a CPU can
run chacha_8block_xor_avx512vl() but not chacha_16block_xor_avx512()
(the choice of vector length is just going to be made by policy).

Given that, could we consolidate these implementations?  I'd suggest:

- Rename the existing chacha-avx512vl-x86_64.S to chacha-avx512-x86_64.S

- Add the CTR16BL rodata, overlapped with CTR8BL such that CTR8BL is
  simply the first 8 entries of CTR16BL.

- Add a macro that expands into either chacha_8block_xor_avx512vl() or
  chacha_16block_xor_avx512(), using a similar approach to
  arch/x86/crypto/aes-xts-avx-x86_64.S.  Use it to generate both.

Does that make sense?

> This will conflict with your cpu_has_xfeatures() removal [1]. Let me know
> if I shall drop the cpu_has_xfeatures(XFEATURE_MASK_AVX512) line.

Well, this is one of the cases where
cpu_has_xfeatures(XFEATURE_MASK_AVX512) was already required by the
existing code (since all AVX512 bits need to be enabled for any EVEX
coded instructions to work, even on xmm/ymm).  But it was never checked,
and no one ever noticed.  I would just continue to leave it out.

You can also drop the check of X86_FEATURE_AVX512F, which is implied by
X86_FEATURE_AVX512VL and X86_FEATURE_AVX512BW already.

- Eric