From: Qu Wenruo <wqu@suse.com>
To: linux-btrfs@vger.kernel.org, linux-crypto@vger.kernel.org
Cc: ebiggers@kernel.org, ardb@kernel.org
Subject: [PATCH RFC v2 0/6] lib/crc: x86: convert the CRC assembly to C intrinsics
Date: Fri, 20 Nov 2026 11:48:58 +1030 [thread overview]
Message-ID: <cover.1795137348.git.wqu@suse.com> (raw)
[CHANGELOG]
v2:
- Use compiler's intrinsics header directly
- Convert all x86 asm files to C
Currently the x86 CRC code is all assembly: the generic [V]PCLMULQDQ
template (12 functions: crc16, crc32 and crc64, three vector lengths each)
and crc32c-3way.S. arm and arm64 already use C intrinsics.
This series converts all of it to C with the compiler's intrinsics, and
removes the assembly. No algorithm, constant or calling convention
changes, so the glue is untouched.
The goal is maintainability and portability, not speed: C is easier to
read, review and extend than the macro-heavy assembly.
And it is also easier to port to userspace tools like btrfs-progs and
xfsprogs which will require optimized CRC for offline operations.
Tests on a Zen5 VM show no obvious change in performance, except for clang
with the MSB-first SSE and AVX2 functions, see below.
Patches:
1: asm/x86-intrinsics.h, a wrapper around <immintrin.h>.
2: crc32c_x86_3way().
3-6: crc32_lsb, crc64_msb, crc64_lsb and crc16_msb, each with the SSE4.1,
AVX2 and AVX512BW/VL variants. The template crc-pclmul-impl.h grows
with them, and the assembly is removed in the last one.
Design decisions and trade-offs
===============================
- ISA selection: target attribute instead of per-file flags
Each function and helper has __attribute__((target(...))), the code is
not built with -msse4.1, -mavx2 or -mavx512*. Per-file flags would
need one object per ISA level, or the SSE function could use AVX-512.
Pros: all vector lengths of a CRC are in one file, and the compiler can
only use an extension in the functions that run after the CPU feature
checks (checked in the objdump).
Cons: no precedent in the kernel for selecting x86 ISA levels this way,
and this is completely compiler dependent. See the GCC 8.1/8.2 issue
below.
- One template, included once per function
The 12 functions differ only in vector length, CRC width and bit order,
so crc-pclmul-impl.h generates one per inclusion, after the includer
defined VL, CRC_BITS, CRC_MSB and CRC_FUNC.
I tried two different solutions:
* Nine standalone functions
About 1400 lines against about 490 with the template.
And any fututre fixes needs to be duplicated 9 times.
* A DEFINE_CRC_PCLMUL_FUNCS() style macro
It can work, but the body cannot have #if, so the wide-vector stages
become constant if()s and GCC warns about the dead code (-Wpsabi).
That macro also makes checkpatch warn about flow control in a macro,
and compiler errors, gdb and perf point at the invocation line.
The include-as-template is not a common practice, but would be the
least-worst solution I can come up with.
- Generated code and performance
Against the assembly (kbuild objects, GCC 16.2, clang 23): clang has
the same total size for the 12 pclmul functions (7453 against 7482
bytes) and the same instruction count in each main loop.
GCC is 18% bigger (SSE +12%, AVX2 +22%, AVX-512 +19%): stack
realignment in every function and, for AVX-512, loads not folded into
vpternlogq (19 instead of 16 instructions in the LSB-first main loop).
crc32c_x86_3way() grows from 400 bytes to 603 (GCC) and 783 (clang),
mostly prologue and unrolling of the tail. The main loop is 15, 17 and
16 instructions (asm, GCC, clang).
Every C function also gets endbr64 and the ftrace call, which the
assembly never had. The call is patched to a nop at boot.
Throughput comes from a userspace harness using the same sources, built
with the kernel's -mno-sse flags, on a Zen5 VM (Ryzen AI 9 HX 370): one
pinned CPU, median of 7 runs. No in-kernel benchmark yet.
Peak is about 7.7, 15.4 and 30.7 GB/s for SSE, AVX2 and AVX-512. GCC
is within 2% of the assembly from 1 KiB, up to 4-6% slower at 64-200
bytes and 8% for AVX2 at 512 bytes.
clang is the same, except crc64_msb and crc16_msb: SSE is 5-8% slower
from 512 bytes, AVX2 5-7% from 1 KiB, but 14% faster at 512 bytes.
AVX-512 is equal.
The clang SSE loop is 175 bytes against 153 for the assembly, as it
keeps data in xmm8-xmm11 (REX prefixes). That is likely the cause,
changing the XOR order did not help.
crc32c_x86_3way() is within 3% of the assembly: 8.6 to 9.0 GB/s at 200
bytes, 14.8 to 14.9 GB/s at 64 KiB.
Known issue: GCC 8.1 and 8.2
GCC 8.1 and 8.2 cannot build the AVX2 functions. Their
<vpclmulqdqintrin.h> declares _mm256_clmulepi64_epi128() with avx512vl
in its target, which fold256() lacks, so it cannot be inlined.
GCC 8.3 and later only need avx. I reproduced it on Compiler Explorer,
where 8.3 to 9.5 build the code cleanly.
Calling __builtin_ia32_vpclmulqdq_v4di() directly for GCC older than 9
fixes it, but it is not included. 8.1 is still the documented minimum,
so: add that workaround, raise the minimum, or ignore 8.1 and 8.2?
Qu Wenruo (6):
x86: Add a wrapper header for the compiler's SIMD intrinsics
lib/crc: x86: Convert crc32c_x86_3way() from assembly to C intrinsics
lib/crc: x86: Convert crc32_lsb_*() from assembly to C
lib/crc: x86: Convert crc64_msb_*() from assembly to C
lib/crc: x86: Convert crc64_lsb_*() from assembly to C
lib/crc: x86: Convert crc16_msb_*() from assembly to C
arch/x86/include/asm/x86-intrinsics.h | 49 +++
lib/crc/Makefile | 11 +-
lib/crc/x86/crc-pclmul-helpers.h | 90 ++++
lib/crc/x86/crc-pclmul-impl.h | 304 ++++++++++++++
lib/crc/x86/crc-pclmul-template.S | 575 --------------------------
lib/crc/x86/crc-pclmul-template.h | 2 +-
lib/crc/x86/crc16-msb-pclmul.S | 6 -
lib/crc/x86/crc16-msb-pclmul.c | 23 ++
lib/crc/x86/crc32-pclmul.S | 6 -
lib/crc/x86/crc32-pclmul.c | 23 ++
lib/crc/x86/crc32.h | 2 +-
lib/crc/x86/crc32c-3way.S | 360 ----------------
lib/crc/x86/crc32c-3way.c | 264 ++++++++++++
lib/crc/x86/crc64-lsb-pclmul.c | 23 ++
lib/crc/x86/crc64-msb-pclmul.c | 23 ++
lib/crc/x86/crc64-pclmul.S | 7 -
16 files changed, 811 insertions(+), 957 deletions(-)
create mode 100644 arch/x86/include/asm/x86-intrinsics.h
create mode 100644 lib/crc/x86/crc-pclmul-helpers.h
create mode 100644 lib/crc/x86/crc-pclmul-impl.h
delete mode 100644 lib/crc/x86/crc-pclmul-template.S
delete mode 100644 lib/crc/x86/crc16-msb-pclmul.S
create mode 100644 lib/crc/x86/crc16-msb-pclmul.c
delete mode 100644 lib/crc/x86/crc32-pclmul.S
create mode 100644 lib/crc/x86/crc32-pclmul.c
delete mode 100644 lib/crc/x86/crc32c-3way.S
create mode 100644 lib/crc/x86/crc32c-3way.c
create mode 100644 lib/crc/x86/crc64-lsb-pclmul.c
create mode 100644 lib/crc/x86/crc64-msb-pclmul.c
delete mode 100644 lib/crc/x86/crc64-pclmul.S
--
2.56.0
next reply other threads:[~2026-11-20 1:19 UTC|newest]
Thread overview: 7+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-20 1:18 Qu Wenruo [this message]
2026-11-20 1:18 ` [PATCH RFC v2 1/6] x86: Add a wrapper header for the compiler's SIMD intrinsics Qu Wenruo
2026-11-20 1:19 ` [PATCH RFC v2 2/6] lib/crc: x86: Convert crc32c_x86_3way() from assembly to C intrinsics Qu Wenruo
2026-11-20 1:19 ` [PATCH RFC v2 3/6] lib/crc: x86: Convert crc32_lsb_*() from assembly to C Qu Wenruo
2026-11-20 1:19 ` [PATCH RFC v2 4/6] lib/crc: x86: Convert crc64_msb_*() " Qu Wenruo
2026-11-20 1:19 ` [PATCH RFC v2 5/6] lib/crc: x86: Convert crc64_lsb_*() " Qu Wenruo
2026-11-20 1:19 ` [PATCH RFC v2 6/6] lib/crc: x86: Convert crc16_msb_*() " Qu Wenruo
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=cover.1795137348.git.wqu@suse.com \
--to=wqu@suse.com \
--cc=ardb@kernel.org \
--cc=ebiggers@kernel.org \
--cc=linux-btrfs@vger.kernel.org \
--cc=linux-crypto@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox