Linux cryptographic layer development
 help / color / mirror / Atom feed
* [PATCH RFC v2 0/6] lib/crc: x86: convert the CRC assembly to C intrinsics
@ 2026-11-20  1:18 Qu Wenruo
  2026-11-20  1:18 ` [PATCH RFC v2 1/6] x86: Add a wrapper header for the compiler's SIMD intrinsics Qu Wenruo
                   ` (6 more replies)
  0 siblings, 7 replies; 11+ messages in thread
From: Qu Wenruo @ 2026-11-20  1:18 UTC (permalink / raw)
  To: linux-btrfs, linux-crypto; +Cc: ebiggers, ardb

[CHANGELOG]
v2:
- Use compiler's intrinsics header directly
- Convert all x86 asm files to C

Currently the x86 CRC code is all assembly: the generic [V]PCLMULQDQ
template (12 functions: crc16, crc32 and crc64, three vector lengths each)
and crc32c-3way.S.  arm and arm64 already use C intrinsics.

This series converts all of it to C with the compiler's intrinsics, and
removes the assembly.  No algorithm, constant or calling convention
changes, so the glue is untouched.

The goal is maintainability and portability, not speed: C is easier to
read, review and extend than the macro-heavy assembly.
And it is also easier to port to userspace tools like btrfs-progs and
xfsprogs which will require optimized CRC for offline operations.

Tests on a Zen5 VM show no obvious change in performance, except for clang
with the MSB-first SSE and AVX2 functions, see below.

Patches:

1:   asm/x86-intrinsics.h, a wrapper around <immintrin.h>.
2:   crc32c_x86_3way().
3-6: crc32_lsb, crc64_msb, crc64_lsb and crc16_msb, each with the SSE4.1,
     AVX2 and AVX512BW/VL variants.  The template crc-pclmul-impl.h grows
     with them, and the assembly is removed in the last one.

Design decisions and trade-offs
===============================

-  ISA selection: target attribute instead of per-file flags

   Each function and helper has __attribute__((target(...))), the code is
   not built with -msse4.1, -mavx2 or -mavx512*.  Per-file flags would
   need one object per ISA level, or the SSE function could use AVX-512.

   Pros: all vector lengths of a CRC are in one file, and the compiler can
   only use an extension in the functions that run after the CPU feature
   checks (checked in the objdump).

   Cons: no precedent in the kernel for selecting x86 ISA levels this way,
   and this is completely compiler dependent. See the GCC 8.1/8.2 issue
   below.

-  One template, included once per function

   The 12 functions differ only in vector length, CRC width and bit order,
   so crc-pclmul-impl.h generates one per inclusion, after the includer
   defined VL, CRC_BITS, CRC_MSB and CRC_FUNC.

   I tried two different solutions:

   * Nine standalone functions
     About 1400 lines against about 490 with the template.
     And any fututre fixes needs to be duplicated 9 times.

   * A DEFINE_CRC_PCLMUL_FUNCS() style macro
     It can work, but the body cannot have #if, so the wide-vector stages
     become constant if()s and GCC warns about the dead code (-Wpsabi).

     That macro also makes checkpatch warn about flow control in a macro,
     and compiler errors, gdb and perf point at the invocation line.

   The include-as-template is not a common practice, but would be the
   least-worst solution I can come up with.

-  Generated code and performance

   Against the assembly (kbuild objects, GCC 16.2, clang 23): clang has
   the same total size for the 12 pclmul functions (7453 against 7482
   bytes) and the same instruction count in each main loop.

   GCC is 18% bigger (SSE +12%, AVX2 +22%, AVX-512 +19%): stack
   realignment in every function and, for AVX-512, loads not folded into
   vpternlogq (19 instead of 16 instructions in the LSB-first main loop).

   crc32c_x86_3way() grows from 400 bytes to 603 (GCC) and 783 (clang),
   mostly prologue and unrolling of the tail.  The main loop is 15, 17 and
   16 instructions (asm, GCC, clang).

   Every C function also gets endbr64 and the ftrace call, which the
   assembly never had.  The call is patched to a nop at boot.

   Throughput comes from a userspace harness using the same sources, built
   with the kernel's -mno-sse flags, on a Zen5 VM (Ryzen AI 9 HX 370): one
   pinned CPU, median of 7 runs.  No in-kernel benchmark yet.

   Peak is about 7.7, 15.4 and 30.7 GB/s for SSE, AVX2 and AVX-512.  GCC
   is within 2% of the assembly from 1 KiB, up to 4-6% slower at 64-200
   bytes and 8% for AVX2 at 512 bytes.

   clang is the same, except crc64_msb and crc16_msb: SSE is 5-8% slower
   from 512 bytes, AVX2 5-7% from 1 KiB, but 14% faster at 512 bytes.
   AVX-512 is equal.

   The clang SSE loop is 175 bytes against 153 for the assembly, as it
   keeps data in xmm8-xmm11 (REX prefixes).  That is likely the cause,
   changing the XOR order did not help.

   crc32c_x86_3way() is within 3% of the assembly: 8.6 to 9.0 GB/s at 200
   bytes, 14.8 to 14.9 GB/s at 64 KiB.

Known issue: GCC 8.1 and 8.2

   GCC 8.1 and 8.2 cannot build the AVX2 functions.  Their
   <vpclmulqdqintrin.h> declares _mm256_clmulepi64_epi128() with avx512vl
   in its target, which fold256() lacks, so it cannot be inlined.

   GCC 8.3 and later only need avx.  I reproduced it on Compiler Explorer,
   where 8.3 to 9.5 build the code cleanly.

   Calling __builtin_ia32_vpclmulqdq_v4di() directly for GCC older than 9
   fixes it, but it is not included.  8.1 is still the documented minimum,
   so: add that workaround, raise the minimum, or ignore 8.1 and 8.2?


Qu Wenruo (6):
  x86: Add a wrapper header for the compiler's SIMD intrinsics
  lib/crc: x86: Convert crc32c_x86_3way() from assembly to C intrinsics
  lib/crc: x86: Convert crc32_lsb_*() from assembly to C
  lib/crc: x86: Convert crc64_msb_*() from assembly to C
  lib/crc: x86: Convert crc64_lsb_*() from assembly to C
  lib/crc: x86: Convert crc16_msb_*() from assembly to C

 arch/x86/include/asm/x86-intrinsics.h |  49 +++
 lib/crc/Makefile                      |  11 +-
 lib/crc/x86/crc-pclmul-helpers.h      |  90 ++++
 lib/crc/x86/crc-pclmul-impl.h         | 304 ++++++++++++++
 lib/crc/x86/crc-pclmul-template.S     | 575 --------------------------
 lib/crc/x86/crc-pclmul-template.h     |   2 +-
 lib/crc/x86/crc16-msb-pclmul.S        |   6 -
 lib/crc/x86/crc16-msb-pclmul.c        |  23 ++
 lib/crc/x86/crc32-pclmul.S            |   6 -
 lib/crc/x86/crc32-pclmul.c            |  23 ++
 lib/crc/x86/crc32.h                   |   2 +-
 lib/crc/x86/crc32c-3way.S             | 360 ----------------
 lib/crc/x86/crc32c-3way.c             | 264 ++++++++++++
 lib/crc/x86/crc64-lsb-pclmul.c        |  23 ++
 lib/crc/x86/crc64-msb-pclmul.c        |  23 ++
 lib/crc/x86/crc64-pclmul.S            |   7 -
 16 files changed, 811 insertions(+), 957 deletions(-)
 create mode 100644 arch/x86/include/asm/x86-intrinsics.h
 create mode 100644 lib/crc/x86/crc-pclmul-helpers.h
 create mode 100644 lib/crc/x86/crc-pclmul-impl.h
 delete mode 100644 lib/crc/x86/crc-pclmul-template.S
 delete mode 100644 lib/crc/x86/crc16-msb-pclmul.S
 create mode 100644 lib/crc/x86/crc16-msb-pclmul.c
 delete mode 100644 lib/crc/x86/crc32-pclmul.S
 create mode 100644 lib/crc/x86/crc32-pclmul.c
 delete mode 100644 lib/crc/x86/crc32c-3way.S
 create mode 100644 lib/crc/x86/crc32c-3way.c
 create mode 100644 lib/crc/x86/crc64-lsb-pclmul.c
 create mode 100644 lib/crc/x86/crc64-msb-pclmul.c
 delete mode 100644 lib/crc/x86/crc64-pclmul.S

-- 
2.56.0


^ permalink raw reply	[flat|nested] 11+ messages in thread

end of thread, other threads:[~2026-11-20  8:42 UTC | newest]

Thread overview: 11+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-11-20  1:18 [PATCH RFC v2 0/6] lib/crc: x86: convert the CRC assembly to C intrinsics Qu Wenruo
2026-11-20  1:18 ` [PATCH RFC v2 1/6] x86: Add a wrapper header for the compiler's SIMD intrinsics Qu Wenruo
2026-11-20  1:19 ` [PATCH RFC v2 2/6] lib/crc: x86: Convert crc32c_x86_3way() from assembly to C intrinsics Qu Wenruo
2026-11-20  1:19 ` [PATCH RFC v2 3/6] lib/crc: x86: Convert crc32_lsb_*() from assembly to C Qu Wenruo
2026-11-20  1:30   ` sashiko-bot
2026-11-20  1:19 ` [PATCH RFC v2 4/6] lib/crc: x86: Convert crc64_msb_*() " Qu Wenruo
2026-11-20  1:19 ` [PATCH RFC v2 5/6] lib/crc: x86: Convert crc64_lsb_*() " Qu Wenruo
2026-11-20  1:19 ` [PATCH RFC v2 6/6] lib/crc: x86: Convert crc16_msb_*() " Qu Wenruo
2026-11-20  3:42 ` [PATCH RFC v2 0/6] lib/crc: x86: convert the CRC assembly to C intrinsics Eric Biggers
2026-11-20  4:24   ` Qu Wenruo
2026-11-20  8:41   ` Peter Zijlstra

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox