* [PATCH RFC v2 1/6] x86: Add a wrapper header for the compiler's SIMD intrinsics
2026-11-20 1:18 [PATCH RFC v2 0/6] lib/crc: x86: convert the CRC assembly to C intrinsics Qu Wenruo
@ 2026-11-20 1:18 ` Qu Wenruo
2026-11-20 1:19 ` [PATCH RFC v2 2/6] lib/crc: x86: Convert crc32c_x86_3way() from assembly to C intrinsics Qu Wenruo
` (5 subsequent siblings)
6 siblings, 0 replies; 11+ messages in thread
From: Qu Wenruo @ 2026-11-20 1:18 UTC (permalink / raw)
To: linux-btrfs, linux-crypto; +Cc: ebiggers, ardb
Allow kernel code to use SSE/AVX/PCLMULQDQ etc. from C through the
compiler's intrinsics, as arm and arm64 already do through
asm/neon-intrinsics.h.
There are some points to notice:
- Workaround for <mm_malloc.h> header
<immintrin.h> pulls in <xmmintrin.h> then <mm_malloc.h> for
_mm_malloc() and mm_free(), which will further pull in <stdlib.h>.
No intrinsic depends on those memory allocation/free helpers.
Workaround by pretending they are already included for both GCC and
clang.
- Skip intrinsics for genksyms
Intrinsics expands to 30~50K lines, but nothing is exported. No need
to expose those for genksyms. This is the same as arm64 NEON wrapper.
Signed-off-by: Qu Wenruo <wqu@suse.com>
---
arch/x86/include/asm/x86-intrinsics.h | 49 +++++++++++++++++++++++++++
1 file changed, 49 insertions(+)
create mode 100644 arch/x86/include/asm/x86-intrinsics.h
diff --git a/arch/x86/include/asm/x86-intrinsics.h b/arch/x86/include/asm/x86-intrinsics.h
new file mode 100644
index 000000000000..694e11b4b3dc
--- /dev/null
+++ b/arch/x86/include/asm/x86-intrinsics.h
@@ -0,0 +1,49 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Wrapper around the compiler's x86 intrinsics headers, so that kernel code
+ * can use e.g. _mm_crc32_u64() and _mm_clmulepi64_si128() from C.
+ *
+ * Modeled after arch/arm64/include/asm/neon-intrinsics.h.
+ *
+ * Users of this header should:
+ *
+ * - Enable the needed ISA extensions per function with
+ * __attribute__((target("..."))), e.g. target("sse4.2,pclmul").
+ * The kernel is built without them (-mno-sse etc.), and this way the
+ * compiler can only use them inside the functions that are meant to
+ * run on CPUs that have them. The helpers calling intrinsics need the
+ * attribute too.
+ *
+ * - Build the object with -isystem $(CC -print-file-name=include).
+ * This is to allow including the compiler's intrinsics headers, which
+ * are not reachable by -nostdinc.
+ */
+
+#ifndef _ASM_X86_X86_INTRINSICS_H
+#define _ASM_X86_X86_INTRINSICS_H
+
+#include <asm-generic/int-ll64.h>
+
+/*
+ * The compiler's <xmmintrin.h> pulls in <mm_malloc.h>, which needs <stdlib.h>
+ * for posix_memalign() and free(). Neither is wanted in the kernel, so pretend
+ * that <mm_malloc.h> was already included.
+ *
+ * The first is for GCC's header, the second for clang's.
+ */
+#ifndef _MM_MALLOC_H_INCLUDED
+#define _MM_MALLOC_H_INCLUDED
+#endif
+#ifndef __MM_MALLOC_H
+#define __MM_MALLOC_H
+#endif
+
+/*
+ * The intrinsics headers expand to 30~50K lines, but nothing exported.
+ * Hide them from genksyms, like the NEON wrapper does.
+ */
+#ifndef __GENKSYMS__
+#include <immintrin.h>
+#endif
+
+#endif /* _ASM_X86_X86_INTRINSICS_H */
--
2.56.0
^ permalink raw reply related [flat|nested] 11+ messages in thread* [PATCH RFC v2 2/6] lib/crc: x86: Convert crc32c_x86_3way() from assembly to C intrinsics
2026-11-20 1:18 [PATCH RFC v2 0/6] lib/crc: x86: convert the CRC assembly to C intrinsics Qu Wenruo
2026-11-20 1:18 ` [PATCH RFC v2 1/6] x86: Add a wrapper header for the compiler's SIMD intrinsics Qu Wenruo
@ 2026-11-20 1:19 ` Qu Wenruo
2026-11-20 1:19 ` [PATCH RFC v2 3/6] lib/crc: x86: Convert crc32_lsb_*() from assembly to C Qu Wenruo
` (4 subsequent siblings)
6 siblings, 0 replies; 11+ messages in thread
From: Qu Wenruo @ 2026-11-20 1:19 UTC (permalink / raw)
To: linux-btrfs, linux-crypto; +Cc: ebiggers, ardb
Rewrite crc32c-3way.S in C using the new asm/x86-intrinsics.h. The
algorithm and the constant table are unchanged: three lanes of up to 128
qwords are CRCed with crc32q, then combined with PCLMULQDQ.
Build changes:
- Enable SSE4.2 and PCLMULQDQ for the function only, with the target
attribute, instead of per-object ISA flags.
- Add -isystem for the compiler's include directory, like arm64 does.
It is in x86-intrin-cflags, for the later conversions to share.
- Use the path-qualified CFLAGS_x86/crc32c-3way.o, as it is in a subdir.
- Drop asmlinkage from the prototype in crc32.h, as it is C now.
- Use #pragma to keep the 4x unrolled loop of 3 crc32_u64() chains
Behavior changes:
- Always handle the unaligned leading part first
Previously it's only for len >= SMALL_SIZE, which means if len <
SMALL_SIZE, we can skip the unaligned handling.
This is not a big deal before, as later instructions can handle
unaligned address, but still doesn't looks good to me.
Short unaligned input will pay up to 7 crc32_u8(), but should be rare.
The license header and authors are kept from the assembly file, and the
SPDX tag now matches its dual license.
Signed-off-by: Qu Wenruo <wqu@suse.com>
---
lib/crc/Makefile | 5 +
lib/crc/x86/crc32.h | 2 +-
lib/crc/x86/crc32c-3way.S | 360 --------------------------------------
lib/crc/x86/crc32c-3way.c | 264 ++++++++++++++++++++++++++++
4 files changed, 270 insertions(+), 361 deletions(-)
delete mode 100644 lib/crc/x86/crc32c-3way.S
create mode 100644 lib/crc/x86/crc32c-3way.c
diff --git a/lib/crc/Makefile b/lib/crc/Makefile
index 386e9c175263..cd7ff35193ad 100644
--- a/lib/crc/Makefile
+++ b/lib/crc/Makefile
@@ -9,6 +9,10 @@ obj-$(CONFIG_CRC16) += crc16.o
obj-$(CONFIG_CRC_CCITT) += crc-ccitt.o
obj-$(CONFIG_CRC_ITU_T) += crc-itu-t.o
+ifdef CONFIG_X86
+x86-intrin-cflags := -isystem $(shell $(CC) -print-file-name=include)
+endif
+
obj-$(CONFIG_CRC_T10DIF) += crc-t10dif.o
crc-t10dif-y := crc-t10dif-main.o
ifeq ($(CONFIG_CRC_T10DIF_ARCH),y)
@@ -32,6 +36,7 @@ crc32-$(CONFIG_S390) += s390/crc32le-vx.o s390/crc32be-vx.o
crc32-$(CONFIG_SPARC) += sparc/crc32c_asm.o
crc32-$(CONFIG_X86) += x86/crc32-pclmul.o
crc32-$(CONFIG_X86_64) += x86/crc32c-3way.o
+CFLAGS_x86/crc32c-3way.o += $(x86-intrin-cflags)
endif
obj-$(CONFIG_CRC64) += crc64.o
diff --git a/lib/crc/x86/crc32.h b/lib/crc/x86/crc32.h
index 19a5e3c6c73b..2b5dd0018576 100644
--- a/lib/crc/x86/crc32.h
+++ b/lib/crc/x86/crc32.h
@@ -34,7 +34,7 @@ static inline u32 crc32_le_arch(u32 crc, const u8 *p, size_t len)
*/
#define CRC32C_PCLMUL_BREAKEVEN 512
-asmlinkage u32 crc32c_x86_3way(u32 crc, const u8 *buffer, size_t len);
+u32 crc32c_x86_3way(u32 crc, const u8 *buffer, size_t len);
static inline u32 crc32c_arch(u32 crc, const u8 *p, size_t len)
{
diff --git a/lib/crc/x86/crc32c-3way.S b/lib/crc/x86/crc32c-3way.S
deleted file mode 100644
index 9b8770503bbc..000000000000
--- a/lib/crc/x86/crc32c-3way.S
+++ /dev/null
@@ -1,360 +0,0 @@
-/*
- * Implement fast CRC32C with PCLMULQDQ instructions. (x86_64)
- *
- * The white papers on CRC32C calculations with PCLMULQDQ instruction can be
- * downloaded from:
- * http://www.intel.com/content/dam/www/public/us/en/documents/white-papers/crc-iscsi-polynomial-crc32-instruction-paper.pdf
- * http://www.intel.com/content/dam/www/public/us/en/documents/white-papers/fast-crc-computation-paper.pdf
- *
- * Copyright (C) 2012 Intel Corporation.
- * Copyright 2024 Google LLC
- *
- * Authors:
- * Wajdi Feghali <wajdi.k.feghali@intel.com>
- * James Guilford <james.guilford@intel.com>
- * David Cote <david.m.cote@intel.com>
- * Tim Chen <tim.c.chen@linux.intel.com>
- *
- * This software is available to you under a choice of one of two
- * licenses. You may choose to be licensed under the terms of the GNU
- * General Public License (GPL) Version 2, available from the file
- * COPYING in the main directory of this source tree, or the
- * OpenIB.org BSD license below:
- *
- * Redistribution and use in source and binary forms, with or
- * without modification, are permitted provided that the following
- * conditions are met:
- *
- * - Redistributions of source code must retain the above
- * copyright notice, this list of conditions and the following
- * disclaimer.
- *
- * - Redistributions in binary form must reproduce the above
- * copyright notice, this list of conditions and the following
- * disclaimer in the documentation and/or other materials
- * provided with the distribution.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
- * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
- * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
- * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
- * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
- * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
- * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
- * SOFTWARE.
- */
-
-#include <linux/linkage.h>
-
-## ISCSI CRC 32 Implementation with crc32 and pclmulqdq Instruction
-
-# Define threshold below which buffers are considered "small" and routed to
-# regular CRC code that does not interleave the CRC instructions.
-#define SMALL_SIZE 200
-
-# u32 crc32c_x86_3way(u32 crc, const u8 *buffer, size_t len);
-
-.text
-SYM_FUNC_START(crc32c_x86_3way)
-#define crc0 %edi
-#define crc0_q %rdi
-#define bufp %rsi
-#define bufp_d %esi
-#define len %rdx
-#define len_dw %edx
-#define n_misaligned %ecx /* overlaps chunk_bytes! */
-#define n_misaligned_q %rcx
-#define chunk_bytes %ecx /* overlaps n_misaligned! */
-#define chunk_bytes_q %rcx
-#define crc1 %r8
-#define crc2 %r9
-
- cmp $SMALL_SIZE, len
- jb .Lsmall
-
- ################################################################
- ## 1) ALIGN:
- ################################################################
- mov bufp_d, n_misaligned
- neg n_misaligned
- and $7, n_misaligned # calculate the misalignment amount of
- # the address
- je .Laligned # Skip if aligned
-
- # Process 1 <= n_misaligned <= 7 bytes individually in order to align
- # the remaining data to an 8-byte boundary.
-.Ldo_align:
- movq (bufp), %rax
- add n_misaligned_q, bufp
- sub n_misaligned_q, len
-.Lalign_loop:
- crc32b %al, crc0 # compute crc32 of 1-byte
- shr $8, %rax # get next byte
- dec n_misaligned
- jne .Lalign_loop
-.Laligned:
-
- ################################################################
- ## 2) PROCESS BLOCK:
- ################################################################
-
- cmp $128*24, len
- jae .Lfull_block
-
-.Lpartial_block:
- # Compute floor(len / 24) to get num qwords to process from each lane.
- imul $2731, len_dw, %eax # 2731 = ceil(2^16 / 24)
- shr $16, %eax
- jmp .Lcrc_3lanes
-
-.Lfull_block:
- # Processing 128 qwords from each lane.
- mov $128, %eax
-
- ################################################################
- ## 3) CRC each of three lanes:
- ################################################################
-
-.Lcrc_3lanes:
- xor crc1,crc1
- xor crc2,crc2
- mov %eax, chunk_bytes
- shl $3, chunk_bytes # num bytes to process from each lane
- sub $5, %eax # 4 for 4x_loop, 1 for special last iter
- jl .Lcrc_3lanes_4x_done
-
- # Unroll the loop by a factor of 4 to reduce the overhead of the loop
- # bookkeeping instructions, which can compete with crc32q for the ALUs.
-.Lcrc_3lanes_4x_loop:
- crc32q (bufp), crc0_q
- crc32q (bufp,chunk_bytes_q), crc1
- crc32q (bufp,chunk_bytes_q,2), crc2
- crc32q 8(bufp), crc0_q
- crc32q 8(bufp,chunk_bytes_q), crc1
- crc32q 8(bufp,chunk_bytes_q,2), crc2
- crc32q 16(bufp), crc0_q
- crc32q 16(bufp,chunk_bytes_q), crc1
- crc32q 16(bufp,chunk_bytes_q,2), crc2
- crc32q 24(bufp), crc0_q
- crc32q 24(bufp,chunk_bytes_q), crc1
- crc32q 24(bufp,chunk_bytes_q,2), crc2
- add $32, bufp
- sub $4, %eax
- jge .Lcrc_3lanes_4x_loop
-
-.Lcrc_3lanes_4x_done:
- add $4, %eax
- jz .Lcrc_3lanes_last_qword
-
-.Lcrc_3lanes_1x_loop:
- crc32q (bufp), crc0_q
- crc32q (bufp,chunk_bytes_q), crc1
- crc32q (bufp,chunk_bytes_q,2), crc2
- add $8, bufp
- dec %eax
- jnz .Lcrc_3lanes_1x_loop
-
-.Lcrc_3lanes_last_qword:
- crc32q (bufp), crc0_q
- crc32q (bufp,chunk_bytes_q), crc1
-# SKIP crc32q (bufp,chunk_bytes_q,2), crc2 ; Don't do this one yet
-
- ################################################################
- ## 4) Combine three results:
- ################################################################
-
- lea (K_table-8)(%rip), %rax # first entry is for idx 1
- pmovzxdq (%rax,chunk_bytes_q), %xmm0 # 2 consts: K1:K2
- lea (chunk_bytes,chunk_bytes,2), %eax # chunk_bytes * 3
- sub %rax, len # len -= chunk_bytes * 3
-
- movq crc0_q, %xmm1 # CRC for block 1
- pclmulqdq $0x00, %xmm0, %xmm1 # Multiply by K2
-
- movq crc1, %xmm2 # CRC for block 2
- pclmulqdq $0x10, %xmm0, %xmm2 # Multiply by K1
-
- pxor %xmm2,%xmm1
- movq %xmm1, %rax
- xor (bufp,chunk_bytes_q,2), %rax
- mov crc2, crc0_q
- crc32 %rax, crc0_q
- lea 8(bufp,chunk_bytes_q,2), bufp
-
- ################################################################
- ## 5) If more blocks remain, goto (2):
- ################################################################
-
- cmp $128*24, len
- jae .Lfull_block
- cmp $SMALL_SIZE, len
- jae .Lpartial_block
-
- #######################################################################
- ## 6) Process any remainder without interleaving:
- #######################################################################
-.Lsmall:
- test len_dw, len_dw
- jz .Ldone
- mov len_dw, %eax
- shr $3, %eax
- jz .Ldo_dword
-.Ldo_qwords:
- crc32q (bufp), crc0_q
- add $8, bufp
- dec %eax
- jnz .Ldo_qwords
-.Ldo_dword:
- test $4, len_dw
- jz .Ldo_word
- crc32l (bufp), crc0
- add $4, bufp
-.Ldo_word:
- test $2, len_dw
- jz .Ldo_byte
- crc32w (bufp), crc0
- add $2, bufp
-.Ldo_byte:
- test $1, len_dw
- jz .Ldone
- crc32b (bufp), crc0
-.Ldone:
- mov crc0, %eax
- RET
-SYM_FUNC_END(crc32c_x86_3way)
-
-.section .rodata, "a", @progbits
- ################################################################
- ## PCLMULQDQ tables
- ## Table is 128 entries x 2 words (8 bytes) each
- ################################################################
-.align 8
-K_table:
- .long 0x493c7d27, 0x00000001
- .long 0xba4fc28e, 0x493c7d27
- .long 0xddc0152b, 0xf20c0dfe
- .long 0x9e4addf8, 0xba4fc28e
- .long 0x39d3b296, 0x3da6d0cb
- .long 0x0715ce53, 0xddc0152b
- .long 0x47db8317, 0x1c291d04
- .long 0x0d3b6092, 0x9e4addf8
- .long 0xc96cfdc0, 0x740eef02
- .long 0x878a92a7, 0x39d3b296
- .long 0xdaece73e, 0x083a6eec
- .long 0xab7aff2a, 0x0715ce53
- .long 0x2162d385, 0xc49f4f67
- .long 0x83348832, 0x47db8317
- .long 0x299847d5, 0x2ad91c30
- .long 0xb9e02b86, 0x0d3b6092
- .long 0x18b33a4e, 0x6992cea2
- .long 0xb6dd949b, 0xc96cfdc0
- .long 0x78d9ccb7, 0x7e908048
- .long 0xbac2fd7b, 0x878a92a7
- .long 0xa60ce07b, 0x1b3d8f29
- .long 0xce7f39f4, 0xdaece73e
- .long 0x61d82e56, 0xf1d0f55e
- .long 0xd270f1a2, 0xab7aff2a
- .long 0xc619809d, 0xa87ab8a8
- .long 0x2b3cac5d, 0x2162d385
- .long 0x65863b64, 0x8462d800
- .long 0x1b03397f, 0x83348832
- .long 0xebb883bd, 0x71d111a8
- .long 0xb3e32c28, 0x299847d5
- .long 0x064f7f26, 0xffd852c6
- .long 0xdd7e3b0c, 0xb9e02b86
- .long 0xf285651c, 0xdcb17aa4
- .long 0x10746f3c, 0x18b33a4e
- .long 0xc7a68855, 0xf37c5aee
- .long 0x271d9844, 0xb6dd949b
- .long 0x8e766a0c, 0x6051d5a2
- .long 0x93a5f730, 0x78d9ccb7
- .long 0x6cb08e5c, 0x18b0d4ff
- .long 0x6b749fb2, 0xbac2fd7b
- .long 0x1393e203, 0x21f3d99c
- .long 0xcec3662e, 0xa60ce07b
- .long 0x96c515bb, 0x8f158014
- .long 0xe6fc4e6a, 0xce7f39f4
- .long 0x8227bb8a, 0xa00457f7
- .long 0xb0cd4768, 0x61d82e56
- .long 0x39c7ff35, 0x8d6d2c43
- .long 0xd7a4825c, 0xd270f1a2
- .long 0x0ab3844b, 0x00ac29cf
- .long 0x0167d312, 0xc619809d
- .long 0xf6076544, 0xe9adf796
- .long 0x26f6a60a, 0x2b3cac5d
- .long 0xa741c1bf, 0x96638b34
- .long 0x98d8d9cb, 0x65863b64
- .long 0x49c3cc9c, 0xe0e9f351
- .long 0x68bce87a, 0x1b03397f
- .long 0x57a3d037, 0x9af01f2d
- .long 0x6956fc3b, 0xebb883bd
- .long 0x42d98888, 0x2cff42cf
- .long 0x3771e98f, 0xb3e32c28
- .long 0xb42ae3d9, 0x88f25a3a
- .long 0x2178513a, 0x064f7f26
- .long 0xe0ac139e, 0x4e36f0b0
- .long 0x170076fa, 0xdd7e3b0c
- .long 0x444dd413, 0xbd6f81f8
- .long 0x6f345e45, 0xf285651c
- .long 0x41d17b64, 0x91c9bd4b
- .long 0xff0dba97, 0x10746f3c
- .long 0xa2b73df1, 0x885f087b
- .long 0xf872e54c, 0xc7a68855
- .long 0x1e41e9fc, 0x4c144932
- .long 0x86d8e4d2, 0x271d9844
- .long 0x651bd98b, 0x52148f02
- .long 0x5bb8f1bc, 0x8e766a0c
- .long 0xa90fd27a, 0xa3c6f37a
- .long 0xb3af077a, 0x93a5f730
- .long 0x4984d782, 0xd7c0557f
- .long 0xca6ef3ac, 0x6cb08e5c
- .long 0x234e0b26, 0x63ded06a
- .long 0xdd66cbbb, 0x6b749fb2
- .long 0x4597456a, 0x4d56973c
- .long 0xe9e28eb4, 0x1393e203
- .long 0x7b3ff57a, 0x9669c9df
- .long 0xc9c8b782, 0xcec3662e
- .long 0x3f70cc6f, 0xe417f38a
- .long 0x93e106a4, 0x96c515bb
- .long 0x62ec6c6d, 0x4b9e0f71
- .long 0xd813b325, 0xe6fc4e6a
- .long 0x0df04680, 0xd104b8fc
- .long 0x2342001e, 0x8227bb8a
- .long 0x0a2a8d7e, 0x5b397730
- .long 0x6d9a4957, 0xb0cd4768
- .long 0xe8b6368b, 0xe78eb416
- .long 0xd2c3ed1a, 0x39c7ff35
- .long 0x995a5724, 0x61ff0e01
- .long 0x9ef68d35, 0xd7a4825c
- .long 0x0c139b31, 0x8d96551c
- .long 0xf2271e60, 0x0ab3844b
- .long 0x0b0bf8ca, 0x0bf80dd2
- .long 0x2664fd8b, 0x0167d312
- .long 0xed64812d, 0x8821abed
- .long 0x02ee03b2, 0xf6076544
- .long 0x8604ae0f, 0x6a45d2b2
- .long 0x363bd6b3, 0x26f6a60a
- .long 0x135c83fd, 0xd8d26619
- .long 0x5fabe670, 0xa741c1bf
- .long 0x35ec3279, 0xde87806c
- .long 0x00bcf5f6, 0x98d8d9cb
- .long 0x8ae00689, 0x14338754
- .long 0x17f27698, 0x49c3cc9c
- .long 0x58ca5f00, 0x5bd2011f
- .long 0xaa7c7ad5, 0x68bce87a
- .long 0xb5cfca28, 0xdd07448e
- .long 0xded288f8, 0x57a3d037
- .long 0x59f229bc, 0xdde8f5b9
- .long 0x6d390dec, 0x6956fc3b
- .long 0x37170390, 0xa3e3e02c
- .long 0x6353c1cc, 0x42d98888
- .long 0xc4584f5c, 0xd73c7bea
- .long 0xf48642e9, 0x3771e98f
- .long 0x531377e2, 0x80ff0093
- .long 0xdd35bc8d, 0xb42ae3d9
- .long 0xb25b29f2, 0x8fe4c34d
- .long 0x9a5ede41, 0x2178513a
- .long 0xa563905d, 0xdf99fc11
- .long 0x45cddf4e, 0xe0ac139e
- .long 0xacfa3103, 0x6c23e841
- .long 0xa51b6135, 0x170076fa
diff --git a/lib/crc/x86/crc32c-3way.c b/lib/crc/x86/crc32c-3way.c
new file mode 100644
index 000000000000..9ed53b97b395
--- /dev/null
+++ b/lib/crc/x86/crc32c-3way.c
@@ -0,0 +1,264 @@
+// SPDX-License-Identifier: GPL-2.0-only OR Linux-OpenIB
+/*
+ * CRC32C using the x86 CRC32 and PCLMULQDQ instructions, via C intrinsics.
+ *
+ * This is a C version of the former crc32c-3way.S. The data is split into
+ * three lanes, which are CRCed independently with crc32q to keep the CPU's
+ * CRC32 unit busy. The three results are then combined using PCLMULQDQ.
+ *
+ * The algorithm is described in:
+ * http://www.intel.com/content/dam/www/public/us/en/documents/white-papers/crc-iscsi-polynomial-crc32-instruction-paper.pdf
+ * http://www.intel.com/content/dam/www/public/us/en/documents/white-papers/fast-crc-computation-paper.pdf
+ *
+ * Copyright (C) 2012 Intel Corporation.
+ * Copyright 2024 Google LLC
+ *
+ * Authors:
+ * Wajdi Feghali <wajdi.k.feghali@intel.com>
+ * James Guilford <james.guilford@intel.com>
+ * David Cote <david.m.cote@intel.com>
+ * Tim Chen <tim.c.chen@linux.intel.com>
+ *
+ * This software is available to you under a choice of one of two
+ * licenses. You may choose to be licensed under the terms of the GNU
+ * General Public License (GPL) Version 2, available from the file
+ * COPYING in the main directory of this source tree, or the
+ * OpenIB.org BSD license below:
+ *
+ * Redistribution and use in source and binary forms, with or
+ * without modification, are permitted provided that the following
+ * conditions are met:
+ *
+ * - Redistributions of source code must retain the above
+ * copyright notice, this list of conditions and the following
+ * disclaimer.
+ *
+ * - Redistributions in binary form must reproduce the above
+ * copyright notice, this list of conditions and the following
+ * disclaimer in the documentation and/or other materials
+ * provided with the distribution.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+ * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+ * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+ * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
+ * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
+ * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+ * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ * SOFTWARE.
+ */
+
+#include <linux/align.h>
+#include <linux/minmax.h>
+#include <linux/types.h>
+#include <asm/x86-intrinsics.h>
+
+__attribute__((target("sse4.2,pclmul")))
+u32 crc32c_x86_3way(u32 crc, const u8 *p, size_t len);
+
+/*
+ * Buffers shorter than this are considered "small" and are processed with a
+ * single stream, as combining the lanes would not pay off.
+ */
+#define SMALL_SIZE 200
+/* The block loop below needs at least 2 qwords per lane. */
+static_assert(SMALL_SIZE / 24 >= 2);
+
+/* The maximum number of qwords processed from each lane per block. */
+#define MAX_LANE_QWORDS 128
+
+/*
+ * Pairs of constants used to combine the lanes, indexed by (lane qwords - 1).
+ * The first constant is multiplied into the first lane's CRC, the second one
+ * into the second lane's CRC. Same values as the former crc32c-3way.S.
+ */
+static const u32 K_table[MAX_LANE_QWORDS][2] = {
+ { 0x493c7d27, 0x00000001 },
+ { 0xba4fc28e, 0x493c7d27 },
+ { 0xddc0152b, 0xf20c0dfe },
+ { 0x9e4addf8, 0xba4fc28e },
+ { 0x39d3b296, 0x3da6d0cb },
+ { 0x0715ce53, 0xddc0152b },
+ { 0x47db8317, 0x1c291d04 },
+ { 0x0d3b6092, 0x9e4addf8 },
+ { 0xc96cfdc0, 0x740eef02 },
+ { 0x878a92a7, 0x39d3b296 },
+ { 0xdaece73e, 0x083a6eec },
+ { 0xab7aff2a, 0x0715ce53 },
+ { 0x2162d385, 0xc49f4f67 },
+ { 0x83348832, 0x47db8317 },
+ { 0x299847d5, 0x2ad91c30 },
+ { 0xb9e02b86, 0x0d3b6092 },
+ { 0x18b33a4e, 0x6992cea2 },
+ { 0xb6dd949b, 0xc96cfdc0 },
+ { 0x78d9ccb7, 0x7e908048 },
+ { 0xbac2fd7b, 0x878a92a7 },
+ { 0xa60ce07b, 0x1b3d8f29 },
+ { 0xce7f39f4, 0xdaece73e },
+ { 0x61d82e56, 0xf1d0f55e },
+ { 0xd270f1a2, 0xab7aff2a },
+ { 0xc619809d, 0xa87ab8a8 },
+ { 0x2b3cac5d, 0x2162d385 },
+ { 0x65863b64, 0x8462d800 },
+ { 0x1b03397f, 0x83348832 },
+ { 0xebb883bd, 0x71d111a8 },
+ { 0xb3e32c28, 0x299847d5 },
+ { 0x064f7f26, 0xffd852c6 },
+ { 0xdd7e3b0c, 0xb9e02b86 },
+ { 0xf285651c, 0xdcb17aa4 },
+ { 0x10746f3c, 0x18b33a4e },
+ { 0xc7a68855, 0xf37c5aee },
+ { 0x271d9844, 0xb6dd949b },
+ { 0x8e766a0c, 0x6051d5a2 },
+ { 0x93a5f730, 0x78d9ccb7 },
+ { 0x6cb08e5c, 0x18b0d4ff },
+ { 0x6b749fb2, 0xbac2fd7b },
+ { 0x1393e203, 0x21f3d99c },
+ { 0xcec3662e, 0xa60ce07b },
+ { 0x96c515bb, 0x8f158014 },
+ { 0xe6fc4e6a, 0xce7f39f4 },
+ { 0x8227bb8a, 0xa00457f7 },
+ { 0xb0cd4768, 0x61d82e56 },
+ { 0x39c7ff35, 0x8d6d2c43 },
+ { 0xd7a4825c, 0xd270f1a2 },
+ { 0x0ab3844b, 0x00ac29cf },
+ { 0x0167d312, 0xc619809d },
+ { 0xf6076544, 0xe9adf796 },
+ { 0x26f6a60a, 0x2b3cac5d },
+ { 0xa741c1bf, 0x96638b34 },
+ { 0x98d8d9cb, 0x65863b64 },
+ { 0x49c3cc9c, 0xe0e9f351 },
+ { 0x68bce87a, 0x1b03397f },
+ { 0x57a3d037, 0x9af01f2d },
+ { 0x6956fc3b, 0xebb883bd },
+ { 0x42d98888, 0x2cff42cf },
+ { 0x3771e98f, 0xb3e32c28 },
+ { 0xb42ae3d9, 0x88f25a3a },
+ { 0x2178513a, 0x064f7f26 },
+ { 0xe0ac139e, 0x4e36f0b0 },
+ { 0x170076fa, 0xdd7e3b0c },
+ { 0x444dd413, 0xbd6f81f8 },
+ { 0x6f345e45, 0xf285651c },
+ { 0x41d17b64, 0x91c9bd4b },
+ { 0xff0dba97, 0x10746f3c },
+ { 0xa2b73df1, 0x885f087b },
+ { 0xf872e54c, 0xc7a68855 },
+ { 0x1e41e9fc, 0x4c144932 },
+ { 0x86d8e4d2, 0x271d9844 },
+ { 0x651bd98b, 0x52148f02 },
+ { 0x5bb8f1bc, 0x8e766a0c },
+ { 0xa90fd27a, 0xa3c6f37a },
+ { 0xb3af077a, 0x93a5f730 },
+ { 0x4984d782, 0xd7c0557f },
+ { 0xca6ef3ac, 0x6cb08e5c },
+ { 0x234e0b26, 0x63ded06a },
+ { 0xdd66cbbb, 0x6b749fb2 },
+ { 0x4597456a, 0x4d56973c },
+ { 0xe9e28eb4, 0x1393e203 },
+ { 0x7b3ff57a, 0x9669c9df },
+ { 0xc9c8b782, 0xcec3662e },
+ { 0x3f70cc6f, 0xe417f38a },
+ { 0x93e106a4, 0x96c515bb },
+ { 0x62ec6c6d, 0x4b9e0f71 },
+ { 0xd813b325, 0xe6fc4e6a },
+ { 0x0df04680, 0xd104b8fc },
+ { 0x2342001e, 0x8227bb8a },
+ { 0x0a2a8d7e, 0x5b397730 },
+ { 0x6d9a4957, 0xb0cd4768 },
+ { 0xe8b6368b, 0xe78eb416 },
+ { 0xd2c3ed1a, 0x39c7ff35 },
+ { 0x995a5724, 0x61ff0e01 },
+ { 0x9ef68d35, 0xd7a4825c },
+ { 0x0c139b31, 0x8d96551c },
+ { 0xf2271e60, 0x0ab3844b },
+ { 0x0b0bf8ca, 0x0bf80dd2 },
+ { 0x2664fd8b, 0x0167d312 },
+ { 0xed64812d, 0x8821abed },
+ { 0x02ee03b2, 0xf6076544 },
+ { 0x8604ae0f, 0x6a45d2b2 },
+ { 0x363bd6b3, 0x26f6a60a },
+ { 0x135c83fd, 0xd8d26619 },
+ { 0x5fabe670, 0xa741c1bf },
+ { 0x35ec3279, 0xde87806c },
+ { 0x00bcf5f6, 0x98d8d9cb },
+ { 0x8ae00689, 0x14338754 },
+ { 0x17f27698, 0x49c3cc9c },
+ { 0x58ca5f00, 0x5bd2011f },
+ { 0xaa7c7ad5, 0x68bce87a },
+ { 0xb5cfca28, 0xdd07448e },
+ { 0xded288f8, 0x57a3d037 },
+ { 0x59f229bc, 0xdde8f5b9 },
+ { 0x6d390dec, 0x6956fc3b },
+ { 0x37170390, 0xa3e3e02c },
+ { 0x6353c1cc, 0x42d98888 },
+ { 0xc4584f5c, 0xd73c7bea },
+ { 0xf48642e9, 0x3771e98f },
+ { 0x531377e2, 0x80ff0093 },
+ { 0xdd35bc8d, 0xb42ae3d9 },
+ { 0xb25b29f2, 0x8fe4c34d },
+ { 0x9a5ede41, 0x2178513a },
+ { 0xa563905d, 0xdf99fc11 },
+ { 0x45cddf4e, 0xe0ac139e },
+ { 0xacfa3103, 0x6c23e841 },
+ { 0xa51b6135, 0x170076fa },
+};
+
+__attribute__((target("sse4.2,pclmul")))
+u32 crc32c_x86_3way(u32 crc, const u8 *p, size_t len)
+{
+ u64 crc0 = crc;
+ /* Align the pointer to 8 bytes, so all the following loads are aligned. */
+ size_t n_misaligned = min((size_t)(PTR_ALIGN(p, 8) - p), len);
+
+ len -= n_misaligned;
+ for (; n_misaligned; n_misaligned--)
+ crc0 = _mm_crc32_u8(crc0, *p++);
+
+ while (len >= SMALL_SIZE) {
+ /* Number of qwords processed from each lane. */
+ const size_t n = min(len / 24, MAX_LANE_QWORDS);
+ const size_t chunk_bytes = n * 8;
+ const u64 *p0 = (const u64 *)p;
+ const u64 *p1 = (const u64 *)(p + chunk_bytes);
+ const u64 *p2 = (const u64 *)(p + chunk_bytes * 2);
+ u64 crc1 = 0, crc2 = 0;
+ __m128i k, x0, x1;
+ size_t i;
+
+#pragma GCC unroll 4
+ for (i = 0; i < n - 1; i++) {
+ crc0 = _mm_crc32_u64(crc0, p0[i]);
+ crc1 = _mm_crc32_u64(crc1, p1[i]);
+ crc2 = _mm_crc32_u64(crc2, p2[i]);
+ }
+ /* The last qword of the third lane is folded in below. */
+ crc0 = _mm_crc32_u64(crc0, p0[i]);
+ crc1 = _mm_crc32_u64(crc1, p1[i]);
+
+ /* Combine the three results. */
+ k = _mm_cvtepu32_epi64(_mm_loadl_epi64((const __m128i *)K_table[n - 1]));
+ x0 = _mm_clmulepi64_si128(_mm_cvtsi64_si128(crc0), k, 0x00);
+ x1 = _mm_clmulepi64_si128(_mm_cvtsi64_si128(crc1), k, 0x10);
+ crc0 = _mm_crc32_u64(crc2,
+ _mm_cvtsi128_si64(_mm_xor_si128(x0, x1)) ^ p2[i]);
+
+ p += chunk_bytes * 3;
+ len -= chunk_bytes * 3;
+ }
+
+ /* Process the remainder without interleaving. */
+ for (; len >= 8; len -= 8, p += 8)
+ crc0 = _mm_crc32_u64(crc0, *(const u64 *)p);
+ if (len & 4) {
+ crc0 = _mm_crc32_u32(crc0, *(const u32 *)p);
+ p += 4;
+ }
+ if (len & 2) {
+ crc0 = _mm_crc32_u16(crc0, *(const u16 *)p);
+ p += 2;
+ }
+ if (len & 1)
+ crc0 = _mm_crc32_u8(crc0, *p);
+
+ return crc0;
+}
--
2.56.0
^ permalink raw reply related [flat|nested] 11+ messages in thread* [PATCH RFC v2 3/6] lib/crc: x86: Convert crc32_lsb_*() from assembly to C
2026-11-20 1:18 [PATCH RFC v2 0/6] lib/crc: x86: convert the CRC assembly to C intrinsics Qu Wenruo
2026-11-20 1:18 ` [PATCH RFC v2 1/6] x86: Add a wrapper header for the compiler's SIMD intrinsics Qu Wenruo
2026-11-20 1:19 ` [PATCH RFC v2 2/6] lib/crc: x86: Convert crc32c_x86_3way() from assembly to C intrinsics Qu Wenruo
@ 2026-11-20 1:19 ` Qu Wenruo
2026-11-20 1:19 ` [PATCH RFC v2 4/6] lib/crc: x86: Convert crc64_msb_*() " Qu Wenruo
` (3 subsequent siblings)
6 siblings, 0 replies; 11+ messages in thread
From: Qu Wenruo @ 2026-11-20 1:19 UTC (permalink / raw)
To: linux-btrfs, linux-crypto; +Cc: ebiggers, ardb
Rewrite the three crc32_lsb functions (PCLMULQDQ with SSE4.1, and
VPCLMULQDQ with AVX2 or AVX512BW/VL) in C using the compiler's
intrinsics, through asm/x86-intrinsics.h.
crc-pclmul-impl.h is the C version of crc-pclmul-template.S: a template,
included once per function with the vector length, CRC width and bit order
defined by the includer. Only what crc32_lsb needs is in it for now.
crc-pclmul-helpers.h has the small helpers: the loads and the fold step
for each vector length.
Each function and helper has a target attribute for the ISA extensions it
needs, so all the vector lengths are in one file. The compiler can only
use those extensions inside the functions that run after the CPU feature
checks.
Build changes:
- crc32-pclmul.c replaces crc32-pclmul.S. It only needs the compiler's
include path, in x86-intrin-cflags.
Known issue: GCC 8.1 and 8.2, the oldest compilers the kernel supports, can
not build the AVX2 function. Their <vpclmulqdqintrin.h> declares
_mm256_clmulepi64_epi128() with avx512vl in its target, which fold256() does
not have, so the call can not be inlined:
error: inlining failed in call to always_inline
'_mm256_clmulepi64_epi128': target specific option mismatch
GCC 8.3 and later only need avx there. Not sure if we still need to
support 8.1 and 8.2.
Signed-off-by: Qu Wenruo <wqu@suse.com>
---
lib/crc/Makefile | 1 +
lib/crc/x86/crc-pclmul-helpers.h | 72 +++++++++
lib/crc/x86/crc-pclmul-impl.h | 258 +++++++++++++++++++++++++++++++
lib/crc/x86/crc32-pclmul.S | 6 -
lib/crc/x86/crc32-pclmul.c | 22 +++
5 files changed, 353 insertions(+), 6 deletions(-)
create mode 100644 lib/crc/x86/crc-pclmul-helpers.h
create mode 100644 lib/crc/x86/crc-pclmul-impl.h
delete mode 100644 lib/crc/x86/crc32-pclmul.S
create mode 100644 lib/crc/x86/crc32-pclmul.c
diff --git a/lib/crc/Makefile b/lib/crc/Makefile
index cd7ff35193ad..baca818f76f7 100644
--- a/lib/crc/Makefile
+++ b/lib/crc/Makefile
@@ -37,6 +37,7 @@ crc32-$(CONFIG_SPARC) += sparc/crc32c_asm.o
crc32-$(CONFIG_X86) += x86/crc32-pclmul.o
crc32-$(CONFIG_X86_64) += x86/crc32c-3way.o
CFLAGS_x86/crc32c-3way.o += $(x86-intrin-cflags)
+CFLAGS_x86/crc32-pclmul.o += $(x86-intrin-cflags)
endif
obj-$(CONFIG_CRC64) += crc64.o
diff --git a/lib/crc/x86/crc-pclmul-helpers.h b/lib/crc/x86/crc-pclmul-helpers.h
new file mode 100644
index 000000000000..c4bac60220f3
--- /dev/null
+++ b/lib/crc/x86/crc-pclmul-helpers.h
@@ -0,0 +1,72 @@
+/* SPDX-License-Identifier: GPL-2.0-or-later */
+/*
+ * Small helpers shared by the C-intrinsics versions of the [V]PCLMULQDQ-based
+ * CRC functions. Each helper has the target attribute for the ISA extensions
+ * it needs, so that they can be used by the functions with the same or a
+ * wider target.
+ *
+ * Copyright 2025 Google LLC
+ *
+ * Author: Eric Biggers <ebiggers@google.com>
+ */
+#ifndef _CRC_PCLMUL_HELPERS_H
+#define _CRC_PCLMUL_HELPERS_H
+
+#include <linux/types.h>
+#include <asm/x86-intrinsics.h>
+
+/* The ISA extensions needed for each vector length */
+#define CRC_TARGET_128 "sse4.1,pclmul"
+#define CRC_TARGET_256 "sse4.1,pclmul,avx2,vpclmulqdq"
+#define CRC_TARGET_512 "sse4.1,pclmul,avx2,vpclmulqdq,avx512f,avx512bw,avx512vl"
+
+/* The target attributes of the helpers, to keep their signatures short */
+#define __crc128 __attribute__((target(CRC_TARGET_128)))
+#define __crc256 __attribute__((target(CRC_TARGET_256)))
+#define __crc512 __attribute__((target(CRC_TARGET_512)))
+
+static __always_inline __crc128 __m128i loadu128(const u8 *p)
+{
+ return _mm_loadu_si128((const __m128i *)p);
+}
+
+/*
+ * Fold acc into data, across the distance that k was generated for: the qwords
+ * of k and acc with the same index are multiplied. The compilers turn the
+ * three-way XOR into a single ternary logic instruction if the target has it.
+ */
+static __always_inline __crc128 __m128i fold128(__m128i acc, __m128i data, __m128i k)
+{
+ __m128i lo = _mm_clmulepi64_si128(acc, k, 0x00);
+ __m128i hi = _mm_clmulepi64_si128(acc, k, 0x11);
+
+ return _mm_xor_si128(_mm_xor_si128(hi, data), lo);
+}
+
+static __always_inline __crc256 __m256i loadu256(const u8 *p)
+{
+ return _mm256_loadu_si256((const __m256i *)p);
+}
+
+static __always_inline __crc256 __m256i fold256(__m256i acc, __m256i data, __m256i k)
+{
+ __m256i lo = _mm256_clmulepi64_epi128(acc, k, 0x00);
+ __m256i hi = _mm256_clmulepi64_epi128(acc, k, 0x11);
+
+ return _mm256_xor_si256(_mm256_xor_si256(hi, data), lo);
+}
+
+static __always_inline __crc512 __m512i loadu512(const u8 *p)
+{
+ return _mm512_loadu_si512(p);
+}
+
+static __always_inline __crc512 __m512i fold512(__m512i acc, __m512i data, __m512i k)
+{
+ __m512i lo = _mm512_clmulepi64_epi128(acc, k, 0x00);
+ __m512i hi = _mm512_clmulepi64_epi128(acc, k, 0x11);
+
+ return _mm512_ternarylogic_epi64(hi, data, lo, 0x96);
+}
+
+#endif /* _CRC_PCLMUL_HELPERS_H */
diff --git a/lib/crc/x86/crc-pclmul-impl.h b/lib/crc/x86/crc-pclmul-impl.h
new file mode 100644
index 000000000000..50fdbadd4b9a
--- /dev/null
+++ b/lib/crc/x86/crc-pclmul-impl.h
@@ -0,0 +1,258 @@
+/* SPDX-License-Identifier: GPL-2.0-or-later */
+/*
+ * Template for the [V]PCLMULQDQ-based CRC functions, using C intrinsics
+ *
+ * This is the C version of the crc-pclmul-template.S that these functions were
+ * generated from. It is a template, not a normal header: each inclusion
+ * generates one function, so a file includes it once for each function, after
+ * the includer has defined:
+ *
+ * VL The vector length in bytes: 16 (PCLMULQDQ with SSE4.1), 32
+ * (VPCLMULQDQ with AVX2) or 64 (VPCLMULQDQ with AVX512BW/VL). The
+ * function gets the matching target attribute.
+ * CRC_BITS The number of bits in the CRC. Currently only 32 is supported.
+ * CRC_FUNC The name of the function to generate. Its prototype is
+ *
+ * crc_t CRC_FUNC(crc_t crc, const u8 *p, size_t len, const void *consts);
+ *
+ * |crc| is the initial CRC, |len| is the data length in bytes, which must be at
+ * least 16, and |consts| points to the fold_across_128_bits_consts field of the
+ * constants generated by scripts/gen-crc-consts.py for the chosen CRC.
+ *
+ * Only least-significant-bit-first CRCs are supported so far.
+ *
+ * VL and CRC_FUNC are undefined at the end of the template, so they have to be
+ * defined again before each inclusion. CRC_BITS is kept, so it can be defined
+ * once for all the inclusions in a file.
+ *
+ * Copyright 2025 Google LLC
+ *
+ * Author: Eric Biggers <ebiggers@google.com>
+ */
+
+#include "crc-pclmul-helpers.h"
+
+#define crc_t u32
+#define CRC_TO_VEC(crc) _mm_cvtsi32_si128(crc)
+
+#if VL == 16
+#define CRC_TARGET CRC_TARGET_128
+#define VEC __m128i
+#define LOG2_VL 4
+#define VLOAD(p) loadu128(p)
+#define VXOR(a, b) _mm_xor_si128(a, b)
+#define VZEXT(x) (x)
+#define VBCAST(x) (x)
+#define VFOLD(a, d, k) fold128(a, d, k)
+#elif VL == 32
+#define CRC_TARGET CRC_TARGET_256
+#define VEC __m256i
+#define LOG2_VL 5
+#define VLOAD(p) loadu256(p)
+#define VXOR(a, b) _mm256_xor_si256(a, b)
+#define VZEXT(x) _mm256_inserti128_si256(_mm256_setzero_si256(), x, 0)
+#define VBCAST(x) _mm256_broadcastsi128_si256(x)
+#define VFOLD(a, d, k) fold256(a, d, k)
+#else
+#define CRC_TARGET CRC_TARGET_512
+#define VEC __m512i
+#define LOG2_VL 6
+#define VLOAD(p) loadu512(p)
+#define VXOR(a, b) _mm512_xor_si512(a, b)
+#define VZEXT(x) _mm512_inserti32x4(_mm512_setzero_si512(), x, 0)
+#define VBCAST(x) _mm512_broadcast_i32x4(x)
+#define VFOLD(a, d, k) fold512(a, d, k)
+#endif /* VL == 16 */
+
+#define LOAD128(p) loadu128(p)
+#define LOAD256(p) loadu256(p)
+/* The constants for folding across 2^i vectors of length VL */
+#define FOLD_CONSTS(i) VBCAST(consts[4 - LOG2_VL - (i)])
+
+__attribute__((target(CRC_TARGET)))
+crc_t CRC_FUNC(crc_t crc, const u8 *p, size_t len, const void *consts_ptr);
+
+/*
+ * |consts_ptr| points to fold_across_128_bits_consts. The fold constants for
+ * 256, 512, 1024 and 2048 bits are right before it. The shuffle table and the
+ * Barrett reduction constants follow it.
+ *
+ * The first vector of data, with the initial CRC XORed into it, is folded into
+ * the next ones, 4 vectors at a time, then 2 and 1 at a time for the data that
+ * remains. Then the accumulator is narrowed down to 128 bits, by folding the
+ * upper half into the lower one, and what remains after that is reduced to the
+ * CRC. The remaining length is tracked by the low bits of |len|, which the
+ * loops do not change: len & VL means one more vector is left after the 4- and
+ * 2-vector steps, and so on.
+ */
+__attribute__((target(CRC_TARGET)))
+crc_t CRC_FUNC(crc_t crc, const u8 *p, size_t len, const void *consts_ptr)
+{
+ const __m128i *consts = consts_ptr;
+ const u8 *shuf_table = (const u8 *)consts + 16;
+ __m128i x0, x1, k128 = consts[0];
+ VEC v0, v1, v2, v3, k;
+#if VL >= 32
+ __m256i y0;
+#endif /* VL >= 32 */
+#if VL == 64
+ __m256i k256;
+#endif /* VL == 64 */
+ size_t rem;
+
+ /* The initial CRC goes in the end holding the high-order coefficients. */
+ x0 = CRC_TO_VEC(crc);
+
+ /* len >= 16 is guaranteed, but not len >= VL. */
+#if VL >= 32
+ if (len < 32) {
+ x0 = _mm_xor_si128(x0, LOAD128(p));
+ p += 16;
+ goto fold_done;
+ }
+#endif /* VL >= 32 */
+#if VL == 64
+ if (len < 64) {
+ y0 = _mm256_xor_si256(
+ _mm256_inserti128_si256(_mm256_setzero_si256(), x0, 0),
+ LOAD256(p));
+ p += 32;
+ goto fold_256_to_128;
+ }
+#endif /* VL == 64 */
+
+ v0 = VXOR(VZEXT(x0), VLOAD(p));
+ if (len < 2 * VL) {
+ p += VL;
+ goto fold_narrow;
+ }
+ if (len < 4 * VL) {
+ v1 = VLOAD(p + VL);
+ p += 2 * VL;
+ goto fold_2_to_1;
+ }
+
+ v1 = VLOAD(p + 1 * VL);
+ v2 = VLOAD(p + 2 * VL);
+ v3 = VLOAD(p + 3 * VL);
+ p += 4 * VL;
+ len -= 4 * VL;
+
+ /* Fold v0-v3 into the next 4 vectors of data. */
+ if (len >= 4 * VL) {
+ k = FOLD_CONSTS(2);
+ do {
+ v0 = VFOLD(v0, VLOAD(p + 0 * VL), k);
+ v1 = VFOLD(v1, VLOAD(p + 1 * VL), k);
+ v2 = VFOLD(v2, VLOAD(p + 2 * VL), k);
+ v3 = VFOLD(v3, VLOAD(p + 3 * VL), k);
+ p += 4 * VL;
+ len -= 4 * VL;
+ } while (len >= 4 * VL);
+ }
+
+ /* Fold v0,v1 into v2,v3, then 2 more vectors if any. */
+ k = FOLD_CONSTS(1);
+ v0 = VFOLD(v0, v2, k);
+ v1 = VFOLD(v1, v3, k);
+ if (len & (2 * VL)) {
+ v0 = VFOLD(v0, VLOAD(p + 0 * VL), k);
+ v1 = VFOLD(v1, VLOAD(p + 1 * VL), k);
+ p += 2 * VL;
+ }
+
+fold_2_to_1:
+ /* Fold v0 into v1, then 1 more vector if any. */
+ k = FOLD_CONSTS(0);
+ v0 = VFOLD(v0, v1, k);
+ if (len & VL) {
+ v0 = VFOLD(v0, VLOAD(p), k);
+ p += VL;
+ }
+
+fold_narrow:
+#if VL == 64
+ /* Fold v0 into 256 bits, then 1 more 256-bit vector if any. */
+ k256 = _mm256_broadcastsi128_si256(consts[-1]);
+ y0 = fold256(_mm512_castsi512_si256(v0),
+ _mm512_extracti64x4_epi64(v0, 1), k256);
+ if (len & 32) {
+ y0 = fold256(y0, LOAD256(p), k256);
+ p += 32;
+ }
+
+fold_256_to_128:
+#endif /* VL == 64 */
+#if VL >= 32
+#if VL == 32
+ y0 = v0;
+#endif /* VL == 32 */
+ /* Fold y0 into 128 bits, then 1 more 128-bit vector if any. */
+ x0 = fold128(_mm256_castsi256_si128(y0),
+ _mm256_extracti128_si256(y0, 1), k128);
+ if (len & 16) {
+ x0 = fold128(x0, LOAD128(p), k128);
+ p += 16;
+ }
+
+fold_done:
+#else
+ x0 = v0;
+#endif /* VL >= 32 */
+ rem = len & 15;
+ if (rem) {
+ /*
+ * 1 <= rem <= 15 data bytes remain. The polynomial is now
+ * A*(x^(8*rem)) + B, where A is the 128-bit polynomial in x0 and
+ * B is the polynomial of the remaining bytes. To avoid needing
+ * fold constants for every rem, rearrange it into
+ * C1*(x^128) + C2, where C1 = floor(A / x^(128 - 8*rem)) and
+ * C2 = A*x^(8*rem) + B mod x^128, then fold C1 into C2. The
+ * shuffle table does the shifts of A: by rem bytes one way for
+ * C2, and by 16 - rem the other way for C1. The last 16 bytes
+ * of the data are B, with A*x^(8*rem) blended in.
+ */
+ const u8 *mask_c2 = shuf_table + 16 + rem;
+ const u8 *mask_c1 = shuf_table + rem;
+ __m128i last16 = LOAD128(p + rem - 16);
+ __m128i mask = loadu128(mask_c2);
+ __m128i c1 = _mm_shuffle_epi8(x0, loadu128(mask_c1));
+ __m128i c2 = _mm_blendv_epi8(_mm_shuffle_epi8(x0, mask),
+ last16, mask);
+
+ x0 = fold128(c1, c2, k128);
+ }
+
+ /*
+ * Compute the CRC as x0 * x^n mod G, where x0 is the 128-bit polynomial
+ * and G is the generator polynomial of the CRC. First multiply by x^n
+ * and reduce to 64 + n bits, using the constant that is also used for
+ * folding across 128 bits. Then do a Barrett reduction in two steps:
+ * multiply by floor(x^(63+n) / G) to get the multiple of G to subtract,
+ * then subtract it. The constants are pre-scaled by x^(64-n) so that
+ * each result lands in a qword that the next pclmulqdq can select.
+ */
+ x1 = _mm_clmulepi64_si128(x0, k128, 0x10);
+ x0 = _mm_xor_si128(_mm_srli_si128(x0, 8), x1);
+ k128 = consts[4];
+ x1 = _mm_clmulepi64_si128(x0, k128, 0x00);
+ x1 = _mm_clmulepi64_si128(x1, k128, 0x10);
+ x0 = _mm_xor_si128(x0, x1);
+ return _mm_extract_epi32(x0, 2);
+}
+
+#undef crc_t
+#undef CRC_TARGET
+#undef VL
+#undef CRC_TO_VEC
+#undef VEC
+#undef LOG2_VL
+#undef VLOAD
+#undef VXOR
+#undef VZEXT
+#undef VBCAST
+#undef VFOLD
+#undef LOAD128
+#undef LOAD256
+#undef FOLD_CONSTS
+#undef CRC_FUNC
diff --git a/lib/crc/x86/crc32-pclmul.S b/lib/crc/x86/crc32-pclmul.S
deleted file mode 100644
index f20f40fb0172..000000000000
--- a/lib/crc/x86/crc32-pclmul.S
+++ /dev/null
@@ -1,6 +0,0 @@
-/* SPDX-License-Identifier: GPL-2.0-or-later */
-// Copyright 2025 Google LLC
-
-#include "crc-pclmul-template.S"
-
-DEFINE_CRC_PCLMUL_FUNCS(crc32_lsb, /* bits= */ 32, /* lsb= */ 1)
diff --git a/lib/crc/x86/crc32-pclmul.c b/lib/crc/x86/crc32-pclmul.c
new file mode 100644
index 000000000000..bb37b0a6d418
--- /dev/null
+++ b/lib/crc/x86/crc32-pclmul.c
@@ -0,0 +1,22 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+/*
+ * CRC-32 using [V]PCLMULQDQ, via C intrinsics
+ *
+ * Copyright 2025 Google LLC
+ */
+
+#include <linux/types.h>
+
+#define CRC_BITS 32
+
+#define VL 16
+#define CRC_FUNC crc32_lsb_pclmul_sse
+#include "crc-pclmul-impl.h"
+
+#define VL 32
+#define CRC_FUNC crc32_lsb_vpclmul_avx2
+#include "crc-pclmul-impl.h"
+
+#define VL 64
+#define CRC_FUNC crc32_lsb_vpclmul_avx512
+#include "crc-pclmul-impl.h"
--
2.56.0
^ permalink raw reply related [flat|nested] 11+ messages in thread* [PATCH RFC v2 4/6] lib/crc: x86: Convert crc64_msb_*() from assembly to C
2026-11-20 1:18 [PATCH RFC v2 0/6] lib/crc: x86: convert the CRC assembly to C intrinsics Qu Wenruo
` (2 preceding siblings ...)
2026-11-20 1:19 ` [PATCH RFC v2 3/6] lib/crc: x86: Convert crc32_lsb_*() from assembly to C Qu Wenruo
@ 2026-11-20 1:19 ` Qu Wenruo
2026-11-20 1:19 ` [PATCH RFC v2 5/6] lib/crc: x86: Convert crc64_lsb_*() " Qu Wenruo
` (2 subsequent siblings)
6 siblings, 0 replies; 11+ messages in thread
From: Qu Wenruo @ 2026-11-20 1:19 UTC (permalink / raw)
To: linux-btrfs, linux-crypto; +Cc: ebiggers, ardb
Rewrite the three crc64_msb functions in C using the template, which now
supports MSB-first CRCs, 64 bits wide.
MSB-first CRCs load the data byte-reflected, with the new load_bswap*()
helpers, put the initial CRC at the other end of the vector, and index the
shuffle table from the other end.
crc64-pclmul.S now only has the crc64_lsb functions.
Build changes:
- Add crc64-msb-pclmul.o, built with x86-intrin-cflags.
Signed-off-by: Qu Wenruo <wqu@suse.com>
---
lib/crc/Makefile | 3 +-
lib/crc/x86/crc-pclmul-helpers.h | 18 +++++++++++
lib/crc/x86/crc-pclmul-impl.h | 52 ++++++++++++++++++++++++--------
lib/crc/x86/crc32-pclmul.c | 1 +
lib/crc/x86/crc64-msb-pclmul.c | 23 ++++++++++++++
lib/crc/x86/crc64-pclmul.S | 1 -
6 files changed, 83 insertions(+), 15 deletions(-)
create mode 100644 lib/crc/x86/crc64-msb-pclmul.c
diff --git a/lib/crc/Makefile b/lib/crc/Makefile
index baca818f76f7..302183401543 100644
--- a/lib/crc/Makefile
+++ b/lib/crc/Makefile
@@ -53,7 +53,8 @@ crc64-$(CONFIG_ARM) += crc64-neon.o
crc64-$(CONFIG_ARM64) += crc64-neon.o
crc64-$(CONFIG_RISCV) += riscv/crc64_lsb.o riscv/crc64_msb.o
-crc64-$(CONFIG_X86) += x86/crc64-pclmul.o
+crc64-$(CONFIG_X86) += x86/crc64-pclmul.o x86/crc64-msb-pclmul.o
+CFLAGS_x86/crc64-msb-pclmul.o += $(x86-intrin-cflags)
endif # CONFIG_CRC64_ARCH
obj-y += tests/
diff --git a/lib/crc/x86/crc-pclmul-helpers.h b/lib/crc/x86/crc-pclmul-helpers.h
index c4bac60220f3..f85deb847cd3 100644
--- a/lib/crc/x86/crc-pclmul-helpers.h
+++ b/lib/crc/x86/crc-pclmul-helpers.h
@@ -30,6 +30,12 @@ static __always_inline __crc128 __m128i loadu128(const u8 *p)
return _mm_loadu_si128((const __m128i *)p);
}
+/* Load 16 bytes, and reflect the bytes within each 128-bit lane. */
+static __always_inline __crc128 __m128i load_bswap128(const u8 *p, __m128i bswap)
+{
+ return _mm_shuffle_epi8(loadu128(p), bswap);
+}
+
/*
* Fold acc into data, across the distance that k was generated for: the qwords
* of k and acc with the same index are multiplied. The compilers turn the
@@ -48,6 +54,12 @@ static __always_inline __crc256 __m256i loadu256(const u8 *p)
return _mm256_loadu_si256((const __m256i *)p);
}
+/* Load 32 bytes, and reflect the bytes within each 128-bit lane. */
+static __always_inline __crc256 __m256i load_bswap256(const u8 *p, __m256i bswap)
+{
+ return _mm256_shuffle_epi8(loadu256(p), bswap);
+}
+
static __always_inline __crc256 __m256i fold256(__m256i acc, __m256i data, __m256i k)
{
__m256i lo = _mm256_clmulepi64_epi128(acc, k, 0x00);
@@ -61,6 +73,12 @@ static __always_inline __crc512 __m512i loadu512(const u8 *p)
return _mm512_loadu_si512(p);
}
+/* Load 64 bytes, and reflect the bytes within each 128-bit lane. */
+static __always_inline __crc512 __m512i load_bswap512(const u8 *p, __m512i bswap)
+{
+ return _mm512_shuffle_epi8(loadu512(p), bswap);
+}
+
static __always_inline __crc512 __m512i fold512(__m512i acc, __m512i data, __m512i k)
{
__m512i lo = _mm512_clmulepi64_epi128(acc, k, 0x00);
diff --git a/lib/crc/x86/crc-pclmul-impl.h b/lib/crc/x86/crc-pclmul-impl.h
index 50fdbadd4b9a..db3d4dcef8ee 100644
--- a/lib/crc/x86/crc-pclmul-impl.h
+++ b/lib/crc/x86/crc-pclmul-impl.h
@@ -11,6 +11,10 @@
* (VPCLMULQDQ with AVX2) or 64 (VPCLMULQDQ with AVX512BW/VL). The
* function gets the matching target attribute.
* CRC_BITS The number of bits in the CRC. Currently only 32 is supported.
+ * CRC_MSB 1 if the CRC processes the most significant bit of each byte
+ * first, i.e. maps bit0 to x^0, ..., bit7 to x^7. 0 if it
+ * processes the least significant bit first, i.e. maps bit0 to x^7,
+ * ..., bit7 to x^0.
* CRC_FUNC The name of the function to generate. Its prototype is
*
* crc_t CRC_FUNC(crc_t crc, const u8 *p, size_t len, const void *consts);
@@ -19,11 +23,9 @@
* least 16, and |consts| points to the fold_across_128_bits_consts field of the
* constants generated by scripts/gen-crc-consts.py for the chosen CRC.
*
- * Only least-significant-bit-first CRCs are supported so far.
- *
* VL and CRC_FUNC are undefined at the end of the template, so they have to be
- * defined again before each inclusion. CRC_BITS is kept, so it can be defined
- * once for all the inclusions in a file.
+ * defined again before each inclusion. CRC_BITS and CRC_MSB are kept, so they
+ * can be defined once for all the inclusions in a file.
*
* Copyright 2025 Google LLC
*
@@ -32,14 +34,19 @@
#include "crc-pclmul-helpers.h"
+#if CRC_BITS == 64
+#define crc_t u64
+#define CRC_TO_VEC(crc) _mm_cvtsi64_si128(crc)
+#else
#define crc_t u32
#define CRC_TO_VEC(crc) _mm_cvtsi32_si128(crc)
+#endif /* CRC_BITS == 64 */
#if VL == 16
#define CRC_TARGET CRC_TARGET_128
#define VEC __m128i
#define LOG2_VL 4
-#define VLOAD(p) loadu128(p)
+#define VLOAD(p) (CRC_MSB ? load_bswap128(p, bswap128) : loadu128(p))
#define VXOR(a, b) _mm_xor_si128(a, b)
#define VZEXT(x) (x)
#define VBCAST(x) (x)
@@ -48,7 +55,7 @@
#define CRC_TARGET CRC_TARGET_256
#define VEC __m256i
#define LOG2_VL 5
-#define VLOAD(p) loadu256(p)
+#define VLOAD(p) (CRC_MSB ? load_bswap256(p, bswap256) : loadu256(p))
#define VXOR(a, b) _mm256_xor_si256(a, b)
#define VZEXT(x) _mm256_inserti128_si256(_mm256_setzero_si256(), x, 0)
#define VBCAST(x) _mm256_broadcastsi128_si256(x)
@@ -57,15 +64,15 @@
#define CRC_TARGET CRC_TARGET_512
#define VEC __m512i
#define LOG2_VL 6
-#define VLOAD(p) loadu512(p)
+#define VLOAD(p) (CRC_MSB ? load_bswap512(p, bswap512) : loadu512(p))
#define VXOR(a, b) _mm512_xor_si512(a, b)
#define VZEXT(x) _mm512_inserti32x4(_mm512_setzero_si512(), x, 0)
#define VBCAST(x) _mm512_broadcast_i32x4(x)
#define VFOLD(a, d, k) fold512(a, d, k)
#endif /* VL == 16 */
-#define LOAD128(p) loadu128(p)
-#define LOAD256(p) loadu256(p)
+#define LOAD128(p) (CRC_MSB ? load_bswap128(p, bswap128) : loadu128(p))
+#define LOAD256(p) (CRC_MSB ? load_bswap256(p, bswap256) : loadu256(p))
/* The constants for folding across 2^i vectors of length VL */
#define FOLD_CONSTS(i) VBCAST(consts[4 - LOG2_VL - (i)])
@@ -74,8 +81,9 @@ crc_t CRC_FUNC(crc_t crc, const u8 *p, size_t len, const void *consts_ptr);
/*
* |consts_ptr| points to fold_across_128_bits_consts. The fold constants for
- * 256, 512, 1024 and 2048 bits are right before it. The shuffle table and the
- * Barrett reduction constants follow it.
+ * 256, 512, 1024 and 2048 bits are right before it, preceded by the byte
+ * reflection mask (MSB-first CRCs only). The shuffle table and the Barrett
+ * reduction constants follow it.
*
* The first vector of data, with the initial CRC XORed into it, is folded into
* the next ones, 4 vectors at a time, then 2 and 1 at a time for the data that
@@ -90,18 +98,23 @@ crc_t CRC_FUNC(crc_t crc, const u8 *p, size_t len, const void *consts_ptr)
{
const __m128i *consts = consts_ptr;
const u8 *shuf_table = (const u8 *)consts + 16;
+ const __m128i bswap128 = CRC_MSB ? consts[-5] : _mm_setzero_si128();
__m128i x0, x1, k128 = consts[0];
VEC v0, v1, v2, v3, k;
#if VL >= 32
+ const __m256i bswap256 = _mm256_broadcastsi128_si256(bswap128);
__m256i y0;
#endif /* VL >= 32 */
#if VL == 64
+ const __m512i bswap512 = _mm512_broadcast_i32x4(bswap128);
__m256i k256;
#endif /* VL == 64 */
size_t rem;
/* The initial CRC goes in the end holding the high-order coefficients. */
x0 = CRC_TO_VEC(crc);
+ if (CRC_MSB)
+ x0 = _mm_slli_si128(x0, (128 - CRC_BITS) / 8);
/* len >= 16 is guaranteed, but not len >= VL. */
#if VL >= 32
@@ -212,8 +225,8 @@ crc_t CRC_FUNC(crc_t crc, const u8 *p, size_t len, const void *consts_ptr)
* C2, and by 16 - rem the other way for C1. The last 16 bytes
* of the data are B, with A*x^(8*rem) blended in.
*/
- const u8 *mask_c2 = shuf_table + 16 + rem;
- const u8 *mask_c1 = shuf_table + rem;
+ const u8 *mask_c2 = shuf_table + (CRC_MSB ? 16 - rem : 16 + rem);
+ const u8 *mask_c1 = shuf_table + (CRC_MSB ? 32 - rem : rem);
__m128i last16 = LOAD128(p + rem - 16);
__m128i mask = loadu128(mask_c2);
__m128i c1 = _mm_shuffle_epi8(x0, loadu128(mask_c1));
@@ -232,6 +245,18 @@ crc_t CRC_FUNC(crc_t crc, const u8 *p, size_t len, const void *consts_ptr)
* then subtract it. The constants are pre-scaled by x^(64-n) so that
* each result lands in a qword that the next pclmulqdq can select.
*/
+#if CRC_MSB
+ x1 = _mm_clmulepi64_si128(x0, k128, 0x01);
+ x0 = _mm_xor_si128(_mm_slli_si128(x0, 8), x1);
+ k128 = consts[4];
+ x1 = _mm_clmulepi64_si128(x0, k128, 0x11);
+ x1 = _mm_xor_si128(x1, x0);
+ x1 = _mm_clmulepi64_si128(x1, k128, 0x01);
+ x0 = _mm_xor_si128(x0, x1);
+#if CRC_BITS == 64
+ return _mm_cvtsi128_si64(x0);
+#endif /* CRC_BITS == 64 */
+#elif CRC_BITS == 32
x1 = _mm_clmulepi64_si128(x0, k128, 0x10);
x0 = _mm_xor_si128(_mm_srli_si128(x0, 8), x1);
k128 = consts[4];
@@ -239,6 +264,7 @@ crc_t CRC_FUNC(crc_t crc, const u8 *p, size_t len, const void *consts_ptr)
x1 = _mm_clmulepi64_si128(x1, k128, 0x10);
x0 = _mm_xor_si128(x0, x1);
return _mm_extract_epi32(x0, 2);
+#endif /* CRC_MSB */
}
#undef crc_t
diff --git a/lib/crc/x86/crc32-pclmul.c b/lib/crc/x86/crc32-pclmul.c
index bb37b0a6d418..3c21db7b0fe1 100644
--- a/lib/crc/x86/crc32-pclmul.c
+++ b/lib/crc/x86/crc32-pclmul.c
@@ -8,6 +8,7 @@
#include <linux/types.h>
#define CRC_BITS 32
+#define CRC_MSB 0
#define VL 16
#define CRC_FUNC crc32_lsb_pclmul_sse
diff --git a/lib/crc/x86/crc64-msb-pclmul.c b/lib/crc/x86/crc64-msb-pclmul.c
new file mode 100644
index 000000000000..2ce4a852b241
--- /dev/null
+++ b/lib/crc/x86/crc64-msb-pclmul.c
@@ -0,0 +1,23 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+/*
+ * CRC-64 (MSB-first) using [V]PCLMULQDQ, via C intrinsics
+ *
+ * Copyright 2025 Google LLC
+ */
+
+#include <linux/types.h>
+
+#define CRC_BITS 64
+#define CRC_MSB 1
+
+#define VL 16
+#define CRC_FUNC crc64_msb_pclmul_sse
+#include "crc-pclmul-impl.h"
+
+#define VL 32
+#define CRC_FUNC crc64_msb_vpclmul_avx2
+#include "crc-pclmul-impl.h"
+
+#define VL 64
+#define CRC_FUNC crc64_msb_vpclmul_avx512
+#include "crc-pclmul-impl.h"
diff --git a/lib/crc/x86/crc64-pclmul.S b/lib/crc/x86/crc64-pclmul.S
index 4173051b5197..9e1230ce5c8f 100644
--- a/lib/crc/x86/crc64-pclmul.S
+++ b/lib/crc/x86/crc64-pclmul.S
@@ -3,5 +3,4 @@
#include "crc-pclmul-template.S"
-DEFINE_CRC_PCLMUL_FUNCS(crc64_msb, /* bits= */ 64, /* lsb= */ 0)
DEFINE_CRC_PCLMUL_FUNCS(crc64_lsb, /* bits= */ 64, /* lsb= */ 1)
--
2.56.0
^ permalink raw reply related [flat|nested] 11+ messages in thread* [PATCH RFC v2 5/6] lib/crc: x86: Convert crc64_lsb_*() from assembly to C
2026-11-20 1:18 [PATCH RFC v2 0/6] lib/crc: x86: convert the CRC assembly to C intrinsics Qu Wenruo
` (3 preceding siblings ...)
2026-11-20 1:19 ` [PATCH RFC v2 4/6] lib/crc: x86: Convert crc64_msb_*() " Qu Wenruo
@ 2026-11-20 1:19 ` Qu Wenruo
2026-11-20 1:19 ` [PATCH RFC v2 6/6] lib/crc: x86: Convert crc16_msb_*() " Qu Wenruo
2026-11-20 3:42 ` [PATCH RFC v2 0/6] lib/crc: x86: convert the CRC assembly to C intrinsics Eric Biggers
6 siblings, 0 replies; 11+ messages in thread
From: Qu Wenruo @ 2026-11-20 1:19 UTC (permalink / raw)
To: linux-btrfs, linux-crypto; +Cc: ebiggers, ardb
Rewrite the three crc64_lsb functions in C using the template, which now
supports the final reduction of LSB-first 64-bit CRCs.
Remove crc64-pclmul.S, as nothing is left in it.
Build changes:
- Add crc64-lsb-pclmul.o, built with x86-intrin-cflags, and drop
crc64-pclmul.o.
Signed-off-by: Qu Wenruo <wqu@suse.com>
---
lib/crc/Makefile | 3 ++-
lib/crc/x86/crc-pclmul-impl.h | 14 ++++++++++++++
lib/crc/x86/crc64-lsb-pclmul.c | 23 +++++++++++++++++++++++
lib/crc/x86/crc64-pclmul.S | 6 ------
4 files changed, 39 insertions(+), 7 deletions(-)
create mode 100644 lib/crc/x86/crc64-lsb-pclmul.c
delete mode 100644 lib/crc/x86/crc64-pclmul.S
diff --git a/lib/crc/Makefile b/lib/crc/Makefile
index 302183401543..343d252a3b33 100644
--- a/lib/crc/Makefile
+++ b/lib/crc/Makefile
@@ -53,8 +53,9 @@ crc64-$(CONFIG_ARM) += crc64-neon.o
crc64-$(CONFIG_ARM64) += crc64-neon.o
crc64-$(CONFIG_RISCV) += riscv/crc64_lsb.o riscv/crc64_msb.o
-crc64-$(CONFIG_X86) += x86/crc64-pclmul.o x86/crc64-msb-pclmul.o
+crc64-$(CONFIG_X86) += x86/crc64-msb-pclmul.o x86/crc64-lsb-pclmul.o
CFLAGS_x86/crc64-msb-pclmul.o += $(x86-intrin-cflags)
+CFLAGS_x86/crc64-lsb-pclmul.o += $(x86-intrin-cflags)
endif # CONFIG_CRC64_ARCH
obj-y += tests/
diff --git a/lib/crc/x86/crc-pclmul-impl.h b/lib/crc/x86/crc-pclmul-impl.h
index db3d4dcef8ee..47b5d75d483b 100644
--- a/lib/crc/x86/crc-pclmul-impl.h
+++ b/lib/crc/x86/crc-pclmul-impl.h
@@ -264,6 +264,20 @@ crc_t CRC_FUNC(crc_t crc, const u8 *p, size_t len, const void *consts_ptr)
x1 = _mm_clmulepi64_si128(x1, k128, 0x10);
x0 = _mm_xor_si128(x0, x1);
return _mm_extract_epi32(x0, 2);
+#else
+ /*
+ * For an LSB-first CRC-64 the low qword of x1, floor(t0 / G), must also
+ * be XORed into the CRC. Move it to the high qword, the one that is
+ * extracted, so that the two XORs can be done together.
+ */
+ x1 = _mm_clmulepi64_si128(x0, k128, 0x10);
+ x0 = _mm_xor_si128(_mm_srli_si128(x0, 8), x1);
+ k128 = consts[4];
+ x1 = _mm_clmulepi64_si128(x0, k128, 0x00);
+ x0 = _mm_xor_si128(x0, _mm_unpacklo_epi64(x1, x1));
+ x1 = _mm_clmulepi64_si128(x1, k128, 0x10);
+ x0 = _mm_xor_si128(x0, x1);
+ return _mm_extract_epi64(x0, 1);
#endif /* CRC_MSB */
}
diff --git a/lib/crc/x86/crc64-lsb-pclmul.c b/lib/crc/x86/crc64-lsb-pclmul.c
new file mode 100644
index 000000000000..a184ffb2f19a
--- /dev/null
+++ b/lib/crc/x86/crc64-lsb-pclmul.c
@@ -0,0 +1,23 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+/*
+ * CRC-64 (LSB-first) using [V]PCLMULQDQ, via C intrinsics
+ *
+ * Copyright 2025 Google LLC
+ */
+
+#include <linux/types.h>
+
+#define CRC_BITS 64
+#define CRC_MSB 0
+
+#define VL 16
+#define CRC_FUNC crc64_lsb_pclmul_sse
+#include "crc-pclmul-impl.h"
+
+#define VL 32
+#define CRC_FUNC crc64_lsb_vpclmul_avx2
+#include "crc-pclmul-impl.h"
+
+#define VL 64
+#define CRC_FUNC crc64_lsb_vpclmul_avx512
+#include "crc-pclmul-impl.h"
diff --git a/lib/crc/x86/crc64-pclmul.S b/lib/crc/x86/crc64-pclmul.S
deleted file mode 100644
index 9e1230ce5c8f..000000000000
--- a/lib/crc/x86/crc64-pclmul.S
+++ /dev/null
@@ -1,6 +0,0 @@
-/* SPDX-License-Identifier: GPL-2.0-or-later */
-// Copyright 2025 Google LLC
-
-#include "crc-pclmul-template.S"
-
-DEFINE_CRC_PCLMUL_FUNCS(crc64_lsb, /* bits= */ 64, /* lsb= */ 1)
--
2.56.0
^ permalink raw reply related [flat|nested] 11+ messages in thread* [PATCH RFC v2 6/6] lib/crc: x86: Convert crc16_msb_*() from assembly to C
2026-11-20 1:18 [PATCH RFC v2 0/6] lib/crc: x86: convert the CRC assembly to C intrinsics Qu Wenruo
` (4 preceding siblings ...)
2026-11-20 1:19 ` [PATCH RFC v2 5/6] lib/crc: x86: Convert crc64_lsb_*() " Qu Wenruo
@ 2026-11-20 1:19 ` Qu Wenruo
2026-11-20 3:42 ` [PATCH RFC v2 0/6] lib/crc: x86: convert the CRC assembly to C intrinsics Eric Biggers
6 siblings, 0 replies; 11+ messages in thread
From: Qu Wenruo @ 2026-11-20 1:19 UTC (permalink / raw)
To: linux-btrfs, linux-crypto; +Cc: ebiggers, ardb
Rewrite the three crc16_msb functions in C using the template, which now
supports 16-bit CRCs.
Remove crc16-msb-pclmul.S, and crc-pclmul-template.S which is not used by
anything any more.
Build changes:
- crc16-msb-pclmul.c replaces crc16-msb-pclmul.S, built with
x86-intrin-cflags.
Signed-off-by: Qu Wenruo <wqu@suse.com>
---
lib/crc/Makefile | 1 +
lib/crc/x86/crc-pclmul-impl.h | 8 +-
lib/crc/x86/crc-pclmul-template.S | 575 ------------------------------
lib/crc/x86/crc-pclmul-template.h | 2 +-
lib/crc/x86/crc16-msb-pclmul.S | 6 -
lib/crc/x86/crc16-msb-pclmul.c | 23 ++
6 files changed, 32 insertions(+), 583 deletions(-)
delete mode 100644 lib/crc/x86/crc-pclmul-template.S
delete mode 100644 lib/crc/x86/crc16-msb-pclmul.S
create mode 100644 lib/crc/x86/crc16-msb-pclmul.c
diff --git a/lib/crc/Makefile b/lib/crc/Makefile
index 343d252a3b33..7a4d628fd221 100644
--- a/lib/crc/Makefile
+++ b/lib/crc/Makefile
@@ -22,6 +22,7 @@ crc-t10dif-$(CONFIG_ARM64) += arm64/crc-t10dif-core.o
crc-t10dif-$(CONFIG_PPC) += powerpc/crct10dif-vpmsum_asm.o
crc-t10dif-$(CONFIG_RISCV) += riscv/crc16_msb.o
crc-t10dif-$(CONFIG_X86) += x86/crc16-msb-pclmul.o
+CFLAGS_x86/crc16-msb-pclmul.o += $(x86-intrin-cflags)
endif
obj-$(CONFIG_CRC32) += crc32.o
diff --git a/lib/crc/x86/crc-pclmul-impl.h b/lib/crc/x86/crc-pclmul-impl.h
index 47b5d75d483b..402fec242037 100644
--- a/lib/crc/x86/crc-pclmul-impl.h
+++ b/lib/crc/x86/crc-pclmul-impl.h
@@ -10,7 +10,8 @@
* VL The vector length in bytes: 16 (PCLMULQDQ with SSE4.1), 32
* (VPCLMULQDQ with AVX2) or 64 (VPCLMULQDQ with AVX512BW/VL). The
* function gets the matching target attribute.
- * CRC_BITS The number of bits in the CRC. Currently only 32 is supported.
+ * CRC_BITS The number of bits in the CRC. Currently 16, 32 and 64 are
+ * supported.
* CRC_MSB 1 if the CRC processes the most significant bit of each byte
* first, i.e. maps bit0 to x^0, ..., bit7 to x^7. 0 if it
* processes the least significant bit first, i.e. maps bit0 to x^7,
@@ -37,6 +38,9 @@
#if CRC_BITS == 64
#define crc_t u64
#define CRC_TO_VEC(crc) _mm_cvtsi64_si128(crc)
+#elif CRC_BITS == 16
+#define crc_t u16
+#define CRC_TO_VEC(crc) _mm_cvtsi32_si128(crc)
#else
#define crc_t u32
#define CRC_TO_VEC(crc) _mm_cvtsi32_si128(crc)
@@ -255,6 +259,8 @@ crc_t CRC_FUNC(crc_t crc, const u8 *p, size_t len, const void *consts_ptr)
x0 = _mm_xor_si128(x0, x1);
#if CRC_BITS == 64
return _mm_cvtsi128_si64(x0);
+#elif CRC_BITS == 16
+ return _mm_extract_epi16(x0, 3);
#endif /* CRC_BITS == 64 */
#elif CRC_BITS == 32
x1 = _mm_clmulepi64_si128(x0, k128, 0x10);
diff --git a/lib/crc/x86/crc-pclmul-template.S b/lib/crc/x86/crc-pclmul-template.S
deleted file mode 100644
index a02f7dc8053e..000000000000
--- a/lib/crc/x86/crc-pclmul-template.S
+++ /dev/null
@@ -1,575 +0,0 @@
-/* SPDX-License-Identifier: GPL-2.0-or-later */
-//
-// Template to generate [V]PCLMULQDQ-based CRC functions for x86
-//
-// Copyright 2025 Google LLC
-//
-// Author: Eric Biggers <ebiggers@google.com>
-
-#include <linux/linkage.h>
-#include <linux/objtool.h>
-
-// Offsets within the generated constants table
-.set OFFSETOF_BSWAP_MASK, -5*16 // msb-first CRCs only
-.set OFFSETOF_FOLD_ACROSS_2048_BITS_CONSTS, -4*16 // must precede next
-.set OFFSETOF_FOLD_ACROSS_1024_BITS_CONSTS, -3*16 // must precede next
-.set OFFSETOF_FOLD_ACROSS_512_BITS_CONSTS, -2*16 // must precede next
-.set OFFSETOF_FOLD_ACROSS_256_BITS_CONSTS, -1*16 // must precede next
-.set OFFSETOF_FOLD_ACROSS_128_BITS_CONSTS, 0*16 // must be 0
-.set OFFSETOF_SHUF_TABLE, 1*16
-.set OFFSETOF_BARRETT_REDUCTION_CONSTS, 4*16
-
-// Emit a VEX (or EVEX) coded instruction if allowed, or emulate it using the
-// corresponding non-VEX instruction plus any needed moves. The supported
-// instruction formats are:
-//
-// - Two-arg [src, dst], where the non-VEX format is the same.
-// - Three-arg [src1, src2, dst] where the non-VEX format is
-// [src1, src2_and_dst]. If src2 != dst, then src1 must != dst too.
-//
-// \insn gives the instruction without a "v" prefix and including any immediate
-// argument if needed to make the instruction follow one of the above formats.
-// If \unaligned_mem_tmp is given, then the emitted non-VEX code moves \arg1 to
-// it first; this is needed when \arg1 is an unaligned mem operand.
-.macro _cond_vex insn:req, arg1:req, arg2:req, arg3, unaligned_mem_tmp
-.if AVX_LEVEL == 0
- // VEX not allowed. Emulate it.
- .ifnb \arg3 // Three-arg [src1, src2, dst]
- .ifc "\arg2", "\arg3" // src2 == dst?
- .ifnb \unaligned_mem_tmp
- movdqu \arg1, \unaligned_mem_tmp
- \insn \unaligned_mem_tmp, \arg3
- .else
- \insn \arg1, \arg3
- .endif
- .else // src2 != dst
- .ifc "\arg1", "\arg3"
- .error "Can't have src1 == dst when src2 != dst"
- .endif
- .ifnb \unaligned_mem_tmp
- movdqu \arg1, \unaligned_mem_tmp
- movdqa \arg2, \arg3
- \insn \unaligned_mem_tmp, \arg3
- .else
- movdqa \arg2, \arg3
- \insn \arg1, \arg3
- .endif
- .endif
- .else // Two-arg [src, dst]
- .ifnb \unaligned_mem_tmp
- movdqu \arg1, \unaligned_mem_tmp
- \insn \unaligned_mem_tmp, \arg2
- .else
- \insn \arg1, \arg2
- .endif
- .endif
-.else
- // VEX is allowed. Emit the desired instruction directly.
- .ifnb \arg3
- v\insn \arg1, \arg2, \arg3
- .else
- v\insn \arg1, \arg2
- .endif
-.endif
-.endm
-
-// Broadcast an aligned 128-bit mem operand to all 128-bit lanes of a vector
-// register of length VL.
-.macro _vbroadcast src, dst
-.if VL == 16
- _cond_vex movdqa, \src, \dst
-.elseif VL == 32
- vbroadcasti128 \src, \dst
-.else
- vbroadcasti32x4 \src, \dst
-.endif
-.endm
-
-// Load \vl bytes from the unaligned mem operand \src into \dst, and if the CRC
-// is msb-first use \bswap_mask to reflect the bytes within each 128-bit lane.
-.macro _load_data vl, src, bswap_mask, dst
-.if \vl < 64
- _cond_vex movdqu, "\src", \dst
-.else
- vmovdqu8 \src, \dst
-.endif
-.if !LSB_CRC
- _cond_vex pshufb, \bswap_mask, \dst, \dst
-.endif
-.endm
-
-.macro _prepare_v0 vl, v0, v1, bswap_mask
-.if LSB_CRC
- .if \vl < 64
- _cond_vex pxor, (BUF), \v0, \v0, unaligned_mem_tmp=\v1
- .else
- vpxorq (BUF), \v0, \v0
- .endif
-.else
- _load_data \vl, (BUF), \bswap_mask, \v1
- .if \vl < 64
- _cond_vex pxor, \v1, \v0, \v0
- .else
- vpxorq \v1, \v0, \v0
- .endif
-.endif
-.endm
-
-// The x^0..x^63 terms, i.e. poly128 mod x^64, i.e. the physically low qword for
-// msb-first order or the physically high qword for lsb-first order
-#define LO64_TERMS 0
-
-// The x^64..x^127 terms, i.e. floor(poly128 / x^64), i.e. the physically high
-// qword for msb-first order or the physically low qword for lsb-first order
-#define HI64_TERMS 1
-
-// Multiply the given \src1_terms of each 128-bit lane of \src1 by the given
-// \src2_terms of each 128-bit lane of \src2, and write the result(s) to \dst.
-.macro _pclmulqdq src1, src1_terms, src2, src2_terms, dst
- _cond_vex "pclmulqdq $((\src1_terms ^ LSB_CRC) << 4) ^ (\src2_terms ^ LSB_CRC),", \
- \src1, \src2, \dst
-.endm
-
-// Fold \acc into \data and store the result back into \acc. \data can be an
-// unaligned mem operand if using VEX is allowed and the CRC is lsb-first so no
-// byte-reflection is needed; otherwise it must be a vector register. \consts
-// is a vector register containing the needed fold constants, and \tmp is a
-// temporary vector register. All arguments must be the same length.
-.macro _fold_vec acc, data, consts, tmp
- _pclmulqdq \consts, HI64_TERMS, \acc, HI64_TERMS, \tmp
- _pclmulqdq \consts, LO64_TERMS, \acc, LO64_TERMS, \acc
-.if AVX_LEVEL <= 2
- _cond_vex pxor, \data, \tmp, \tmp
- _cond_vex pxor, \tmp, \acc, \acc
-.else
- vpternlogq $0x96, \data, \tmp, \acc
-.endif
-.endm
-
-// Fold \acc into \data and store the result back into \acc. \data is an
-// unaligned mem operand, \consts is a vector register containing the needed
-// fold constants, \bswap_mask is a vector register containing the
-// byte-reflection table if the CRC is msb-first, and \tmp1 and \tmp2 are
-// temporary vector registers. All arguments must have length \vl.
-.macro _fold_vec_mem vl, acc, data, consts, bswap_mask, tmp1, tmp2
-.if AVX_LEVEL == 0 || !LSB_CRC
- _load_data \vl, \data, \bswap_mask, \tmp1
- _fold_vec \acc, \tmp1, \consts, \tmp2
-.else
- _fold_vec \acc, \data, \consts, \tmp1
-.endif
-.endm
-
-// Load the constants for folding across 2**i vectors of length VL at a time
-// into all 128-bit lanes of the vector register CONSTS.
-.macro _load_vec_folding_consts i
- _vbroadcast OFFSETOF_FOLD_ACROSS_128_BITS_CONSTS+(4-LOG2_VL-\i)*16(CONSTS_PTR), \
- CONSTS
-.endm
-
-// Given vector registers \v0 and \v1 of length \vl, fold \v0 into \v1 and store
-// the result back into \v0. If the remaining length mod \vl is nonzero, also
-// fold \vl data bytes from BUF. For both operations the fold distance is \vl.
-// \consts must be a register of length \vl containing the fold constants.
-.macro _fold_vec_final vl, v0, v1, consts, bswap_mask, tmp1, tmp2
- _fold_vec \v0, \v1, \consts, \tmp1
- test $\vl, LEN8
- jz .Lfold_vec_final_done\@
- _fold_vec_mem \vl, \v0, (BUF), \consts, \bswap_mask, \tmp1, \tmp2
- add $\vl, BUF
-.Lfold_vec_final_done\@:
-.endm
-
-// This macro generates the body of a CRC function with the following prototype:
-//
-// crc_t crc_func(crc_t crc, const u8 *buf, size_t len, const void *consts);
-//
-// |crc| is the initial CRC, and crc_t is a data type wide enough to hold it.
-// |buf| is the data to checksum. |len| is the data length in bytes, which must
-// be at least 16. |consts| is a pointer to the fold_across_128_bits_consts
-// field of the constants struct that was generated for the chosen CRC variant.
-//
-// Moving onto the macro parameters, \n is the number of bits in the CRC, e.g.
-// 32 for a CRC-32. Currently the supported values are 8, 16, 32, and 64. If
-// the file is compiled in i386 mode, then the maximum supported value is 32.
-//
-// \lsb_crc is 1 if the CRC processes the least significant bit of each byte
-// first, i.e. maps bit0 to x^7, bit1 to x^6, ..., bit7 to x^0. \lsb_crc is 0
-// if the CRC processes the most significant bit of each byte first, i.e. maps
-// bit0 to x^0, bit1 to x^1, bit7 to x^7.
-//
-// \vl is the maximum length of vector register to use in bytes: 16, 32, or 64.
-//
-// \avx_level is the level of AVX support to use: 0 for SSE only, 2 for AVX2, or
-// 512 for AVX512.
-//
-// If \vl == 16 && \avx_level == 0, the generated code requires:
-// PCLMULQDQ && SSE4.1. (Note: all known CPUs with PCLMULQDQ also have SSE4.1.)
-//
-// If \vl == 32 && \avx_level == 2, the generated code requires:
-// VPCLMULQDQ && AVX2.
-//
-// If \vl == 64 && \avx_level == 512, the generated code requires:
-// VPCLMULQDQ && AVX512BW && AVX512VL.
-//
-// Other \vl and \avx_level combinations are either not supported or not useful.
-.macro _crc_pclmul n, lsb_crc, vl, avx_level
- .set LSB_CRC, \lsb_crc
- .set VL, \vl
- .set AVX_LEVEL, \avx_level
-
- // Define aliases for the xmm, ymm, or zmm registers according to VL.
-.irp i, 0,1,2,3,4,5,6,7
- .if VL == 16
- .set V\i, %xmm\i
- .set LOG2_VL, 4
- .elseif VL == 32
- .set V\i, %ymm\i
- .set LOG2_VL, 5
- .elseif VL == 64
- .set V\i, %zmm\i
- .set LOG2_VL, 6
- .else
- .error "Unsupported vector length"
- .endif
-.endr
- // Define aliases for the function parameters.
- // Note: when crc_t is shorter than u32, zero-extension to 32 bits is
- // guaranteed by the ABI. Zero-extension to 64 bits is *not* guaranteed
- // when crc_t is shorter than u64.
-#ifdef __x86_64__
-.if \n <= 32
- .set CRC, %edi
-.else
- .set CRC, %rdi
-.endif
- .set BUF, %rsi
- .set LEN, %rdx
- .set LEN32, %edx
- .set LEN8, %dl
- .set CONSTS_PTR, %rcx
-#else
- // 32-bit support, assuming -mregparm=3 and not including support for
- // CRC-64 (which would use both eax and edx to pass the crc parameter).
- .set CRC, %eax
- .set BUF, %edx
- .set LEN, %ecx
- .set LEN32, %ecx
- .set LEN8, %cl
- .set CONSTS_PTR, %ebx // Passed on stack
-#endif
-
- // Define aliases for some local variables. V0-V5 are used without
- // aliases (for accumulators, data, temporary values, etc). Staying
- // within the first 8 vector registers keeps the code 32-bit SSE
- // compatible and reduces the size of 64-bit SSE code slightly.
- .set BSWAP_MASK, V6
- .set BSWAP_MASK_YMM, %ymm6
- .set BSWAP_MASK_XMM, %xmm6
- .set CONSTS, V7
- .set CONSTS_YMM, %ymm7
- .set CONSTS_XMM, %xmm7
-
- // Use ANNOTATE_NOENDBR to suppress an objtool warning, since the
- // functions generated by this macro are called only by static_call.
- ANNOTATE_NOENDBR
-
-#ifdef __i386__
- push CONSTS_PTR
- mov 8(%esp), CONSTS_PTR
-#endif
-
- // Create a 128-bit vector that contains the initial CRC in the end
- // representing the high-order polynomial coefficients, and the rest 0.
- // If the CRC is msb-first, also load the byte-reflection table.
-.if \n <= 32
- _cond_vex movd, CRC, %xmm0
-.else
- _cond_vex movq, CRC, %xmm0
-.endif
-.if !LSB_CRC
- _cond_vex pslldq, $(128-\n)/8, %xmm0, %xmm0
- _vbroadcast OFFSETOF_BSWAP_MASK(CONSTS_PTR), BSWAP_MASK
-.endif
-
- // Load the first vector of data and XOR the initial CRC into the
- // appropriate end of the first 128-bit lane of data. If LEN < VL, then
- // use a short vector and jump ahead to the final reduction. (LEN >= 16
- // is guaranteed here but not necessarily LEN >= VL.)
-.if VL >= 32
- cmp $VL, LEN
- jae .Lat_least_1vec\@
- .if VL == 64
- cmp $32, LEN32
- jb .Lless_than_32bytes\@
- _prepare_v0 32, %ymm0, %ymm1, BSWAP_MASK_YMM
- add $32, BUF
- jmp .Lreduce_256bits_to_128bits\@
-.Lless_than_32bytes\@:
- .endif
- _prepare_v0 16, %xmm0, %xmm1, BSWAP_MASK_XMM
- add $16, BUF
- vmovdqa OFFSETOF_FOLD_ACROSS_128_BITS_CONSTS(CONSTS_PTR), CONSTS_XMM
- jmp .Lcheck_for_partial_block\@
-.Lat_least_1vec\@:
-.endif
- _prepare_v0 VL, V0, V1, BSWAP_MASK
-
- // Handle VL <= LEN < 4*VL.
- cmp $4*VL-1, LEN
- ja .Lat_least_4vecs\@
- add $VL, BUF
- // If VL <= LEN < 2*VL, then jump ahead to the reduction from 1 vector.
- // If VL==16 then load fold_across_128_bits_consts first, as the final
- // reduction depends on it and it won't be loaded anywhere else.
- cmp $2*VL-1, LEN32
-.if VL == 16
- _cond_vex movdqa, OFFSETOF_FOLD_ACROSS_128_BITS_CONSTS(CONSTS_PTR), CONSTS_XMM
-.endif
- jbe .Lreduce_1vec_to_128bits\@
- // Otherwise 2*VL <= LEN < 4*VL. Load one more vector and jump ahead to
- // the reduction from 2 vectors.
- _load_data VL, (BUF), BSWAP_MASK, V1
- add $VL, BUF
- jmp .Lreduce_2vecs_to_1\@
-
-.Lat_least_4vecs\@:
- // Load 3 more vectors of data.
- _load_data VL, 1*VL(BUF), BSWAP_MASK, V1
- _load_data VL, 2*VL(BUF), BSWAP_MASK, V2
- _load_data VL, 3*VL(BUF), BSWAP_MASK, V3
- sub $-4*VL, BUF // Shorter than 'add 4*VL' when VL=32
- add $-4*VL, LEN // Shorter than 'sub 4*VL' when VL=32
-
- // Main loop: while LEN >= 4*VL, fold the 4 vectors V0-V3 into the next
- // 4 vectors of data and write the result back to V0-V3.
- cmp $4*VL-1, LEN // Shorter than 'cmp 4*VL' when VL=32
- jbe .Lreduce_4vecs_to_2\@
- _load_vec_folding_consts 2
-.Lfold_4vecs_loop\@:
- _fold_vec_mem VL, V0, 0*VL(BUF), CONSTS, BSWAP_MASK, V4, V5
- _fold_vec_mem VL, V1, 1*VL(BUF), CONSTS, BSWAP_MASK, V4, V5
- _fold_vec_mem VL, V2, 2*VL(BUF), CONSTS, BSWAP_MASK, V4, V5
- _fold_vec_mem VL, V3, 3*VL(BUF), CONSTS, BSWAP_MASK, V4, V5
- sub $-4*VL, BUF
- add $-4*VL, LEN
- cmp $4*VL-1, LEN
- ja .Lfold_4vecs_loop\@
-
- // Fold V0,V1 into V2,V3 and write the result back to V0,V1. Then fold
- // two more vectors of data from BUF, if at least that much remains.
-.Lreduce_4vecs_to_2\@:
- _load_vec_folding_consts 1
- _fold_vec V0, V2, CONSTS, V4
- _fold_vec V1, V3, CONSTS, V4
- test $2*VL, LEN8
- jz .Lreduce_2vecs_to_1\@
- _fold_vec_mem VL, V0, 0*VL(BUF), CONSTS, BSWAP_MASK, V4, V5
- _fold_vec_mem VL, V1, 1*VL(BUF), CONSTS, BSWAP_MASK, V4, V5
- sub $-2*VL, BUF
-
- // Fold V0 into V1 and write the result back to V0. Then fold one more
- // vector of data from BUF, if at least that much remains.
-.Lreduce_2vecs_to_1\@:
- _load_vec_folding_consts 0
- _fold_vec_final VL, V0, V1, CONSTS, BSWAP_MASK, V4, V5
-
-.Lreduce_1vec_to_128bits\@:
-.if VL == 64
- // Reduce 512-bit %zmm0 to 256-bit %ymm0. Then fold 256 more bits of
- // data from BUF, if at least that much remains.
- vbroadcasti128 OFFSETOF_FOLD_ACROSS_256_BITS_CONSTS(CONSTS_PTR), CONSTS_YMM
- vextracti64x4 $1, %zmm0, %ymm1
- _fold_vec_final 32, %ymm0, %ymm1, CONSTS_YMM, BSWAP_MASK_YMM, %ymm4, %ymm5
-.Lreduce_256bits_to_128bits\@:
-.endif
-.if VL >= 32
- // Reduce 256-bit %ymm0 to 128-bit %xmm0. Then fold 128 more bits of
- // data from BUF, if at least that much remains.
- vmovdqa OFFSETOF_FOLD_ACROSS_128_BITS_CONSTS(CONSTS_PTR), CONSTS_XMM
- vextracti128 $1, %ymm0, %xmm1
- _fold_vec_final 16, %xmm0, %xmm1, CONSTS_XMM, BSWAP_MASK_XMM, %xmm4, %xmm5
-.Lcheck_for_partial_block\@:
-.endif
- and $15, LEN32
- jz .Lreduce_128bits_to_crc\@
-
- // 1 <= LEN <= 15 data bytes remain in BUF. The polynomial is now
- // A*(x^(8*LEN)) + B, where A is the 128-bit polynomial stored in %xmm0
- // and B is the polynomial of the remaining LEN data bytes. To reduce
- // this to 128 bits without needing fold constants for each possible
- // LEN, rearrange this expression into C1*(x^128) + C2, where
- // C1 = floor(A / x^(128 - 8*LEN)) and C2 = A*x^(8*LEN) + B mod x^128.
- // Then fold C1 into C2, which is just another fold across 128 bits.
-
-.if !LSB_CRC || AVX_LEVEL == 0
- // Load the last 16 data bytes. Note that originally LEN was >= 16.
- _load_data 16, "-16(BUF,LEN)", BSWAP_MASK_XMM, %xmm2
-.endif // Else will use vpblendvb mem operand later.
-.if !LSB_CRC
- neg LEN // Needed for indexing shuf_table
-.endif
-
- // tmp = A*x^(8*LEN) mod x^128
- // lsb: pshufb by [LEN, LEN+1, ..., 15, -1, -1, ..., -1]
- // i.e. right-shift by LEN bytes.
- // msb: pshufb by [-1, -1, ..., -1, 0, 1, ..., 15-LEN]
- // i.e. left-shift by LEN bytes.
- _cond_vex movdqu, "OFFSETOF_SHUF_TABLE+16(CONSTS_PTR,LEN)", %xmm3
- _cond_vex pshufb, %xmm3, %xmm0, %xmm1
-
- // C1 = floor(A / x^(128 - 8*LEN))
- // lsb: pshufb by [-1, -1, ..., -1, 0, 1, ..., LEN-1]
- // i.e. left-shift by 16-LEN bytes.
- // msb: pshufb by [16-LEN, 16-LEN+1, ..., 15, -1, -1, ..., -1]
- // i.e. right-shift by 16-LEN bytes.
- _cond_vex pshufb, "OFFSETOF_SHUF_TABLE+32*!LSB_CRC(CONSTS_PTR,LEN)", \
- %xmm0, %xmm0, unaligned_mem_tmp=%xmm4
-
- // C2 = tmp + B. This is just a blend of tmp with the last 16 data
- // bytes (reflected if msb-first). The blend mask is the shuffle table
- // that was used to create tmp. 0 selects tmp, and 1 last16databytes.
-.if AVX_LEVEL == 0
- movdqa %xmm0, %xmm4
- movdqa %xmm3, %xmm0
- pblendvb %xmm2, %xmm1 // uses %xmm0 as implicit operand
- movdqa %xmm4, %xmm0
-.elseif LSB_CRC
- vpblendvb %xmm3, -16(BUF,LEN), %xmm1, %xmm1
-.else
- vpblendvb %xmm3, %xmm2, %xmm1, %xmm1
-.endif
-
- // Fold C1 into C2 and store the 128-bit result in %xmm0.
- _fold_vec %xmm0, %xmm1, CONSTS_XMM, %xmm4
-
-.Lreduce_128bits_to_crc\@:
- // Compute the CRC as %xmm0 * x^n mod G. Here %xmm0 means the 128-bit
- // polynomial stored in %xmm0 (using either lsb-first or msb-first bit
- // order according to LSB_CRC), and G is the CRC's generator polynomial.
-
- // First, multiply %xmm0 by x^n and reduce the result to 64+n bits:
- //
- // t0 := (x^(64+n) mod G) * floor(%xmm0 / x^64) +
- // x^n * (%xmm0 mod x^64)
- //
- // Store t0 * x^(64-n) in %xmm0. I.e., actually do:
- //
- // %xmm0 := ((x^(64+n) mod G) * x^(64-n)) * floor(%xmm0 / x^64) +
- // x^64 * (%xmm0 mod x^64)
- //
- // The extra unreduced factor of x^(64-n) makes floor(t0 / x^n) aligned
- // to the HI64_TERMS of %xmm0 so that the next pclmulqdq can easily
- // select it. The 64-bit constant (x^(64+n) mod G) * x^(64-n) in the
- // msb-first case, or (x^(63+n) mod G) * x^(64-n) in the lsb-first case
- // (considering the extra factor of x that gets implicitly introduced by
- // each pclmulqdq when using lsb-first order), is identical to the
- // constant that was used earlier for folding the LO64_TERMS across 128
- // bits. Thus it's already available in LO64_TERMS of CONSTS_XMM.
- _pclmulqdq CONSTS_XMM, LO64_TERMS, %xmm0, HI64_TERMS, %xmm1
-.if LSB_CRC
- _cond_vex psrldq, $8, %xmm0, %xmm0 // x^64 * (%xmm0 mod x^64)
-.else
- _cond_vex pslldq, $8, %xmm0, %xmm0 // x^64 * (%xmm0 mod x^64)
-.endif
- _cond_vex pxor, %xmm1, %xmm0, %xmm0
- // The HI64_TERMS of %xmm0 now contain floor(t0 / x^n).
- // The LO64_TERMS of %xmm0 now contain (t0 mod x^n) * x^(64-n).
-
- // First step of Barrett reduction: Compute floor(t0 / G). This is the
- // polynomial by which G needs to be multiplied to cancel out the x^n
- // and higher terms of t0, i.e. to reduce t0 mod G. First do:
- //
- // t1 := floor(x^(63+n) / G) * x * floor(t0 / x^n)
- //
- // Then the desired value floor(t0 / G) is floor(t1 / x^64). The 63 in
- // x^(63+n) is the maximum degree of floor(t0 / x^n) and thus the lowest
- // value that makes enough precision be carried through the calculation.
- //
- // The '* x' makes it so the result is floor(t1 / x^64) rather than
- // floor(t1 / x^63), making it qword-aligned in HI64_TERMS so that it
- // can be extracted much more easily in the next step. In the lsb-first
- // case the '* x' happens implicitly. In the msb-first case it must be
- // done explicitly; floor(x^(63+n) / G) * x is a 65-bit constant, so the
- // constant passed to pclmulqdq is (floor(x^(63+n) / G) * x) - x^64, and
- // the multiplication by the x^64 term is handled using a pxor. The
- // pxor causes the low 64 terms of t1 to be wrong, but they are unused.
- _cond_vex movdqa, OFFSETOF_BARRETT_REDUCTION_CONSTS(CONSTS_PTR), CONSTS_XMM
- _pclmulqdq CONSTS_XMM, HI64_TERMS, %xmm0, HI64_TERMS, %xmm1
-.if !LSB_CRC
- _cond_vex pxor, %xmm0, %xmm1, %xmm1 // += x^64 * floor(t0 / x^n)
-.endif
- // The HI64_TERMS of %xmm1 now contain floor(t1 / x^64) = floor(t0 / G).
-
- // Second step of Barrett reduction: Cancel out the x^n and higher terms
- // of t0 by subtracting the needed multiple of G. This gives the CRC:
- //
- // crc := t0 - (G * floor(t0 / G))
- //
- // But %xmm0 contains t0 * x^(64-n), so it's more convenient to do:
- //
- // crc := ((t0 * x^(64-n)) - ((G * x^(64-n)) * floor(t0 / G))) / x^(64-n)
- //
- // Furthermore, since the resulting CRC is n-bit, if mod x^n is
- // explicitly applied to it then the x^n term of G makes no difference
- // in the result and can be omitted. This helps keep the constant
- // multiplier in 64 bits in most cases. This gives the following:
- //
- // %xmm0 := %xmm0 - (((G - x^n) * x^(64-n)) * floor(t0 / G))
- // crc := (%xmm0 / x^(64-n)) mod x^n
- //
- // In the lsb-first case, each pclmulqdq implicitly introduces
- // an extra factor of x, so in that case the constant that needs to be
- // passed to pclmulqdq is actually '(G - x^n) * x^(63-n)' when n <= 63.
- // For lsb-first CRCs where n=64, the extra factor of x cannot be as
- // easily avoided. In that case, instead pass '(G - x^n - x^0) / x' to
- // pclmulqdq and handle the x^0 term (i.e. 1) separately. (All CRC
- // polynomials have nonzero x^n and x^0 terms.) It works out as: the
- // CRC has be XORed with the physically low qword of %xmm1, representing
- // floor(t0 / G). The most efficient way to do that is to move it to
- // the physically high qword and use a ternlog to combine the two XORs.
-.if LSB_CRC && \n == 64
- _cond_vex punpcklqdq, %xmm1, %xmm2, %xmm2
- _pclmulqdq CONSTS_XMM, LO64_TERMS, %xmm1, HI64_TERMS, %xmm1
- .if AVX_LEVEL <= 2
- _cond_vex pxor, %xmm2, %xmm0, %xmm0
- _cond_vex pxor, %xmm1, %xmm0, %xmm0
- .else
- vpternlogq $0x96, %xmm2, %xmm1, %xmm0
- .endif
- _cond_vex "pextrq $1,", %xmm0, %rax // (%xmm0 / x^0) mod x^64
-.else
- _pclmulqdq CONSTS_XMM, LO64_TERMS, %xmm1, HI64_TERMS, %xmm1
- _cond_vex pxor, %xmm1, %xmm0, %xmm0
- .if \n == 8
- _cond_vex "pextrb $7 + LSB_CRC,", %xmm0, %eax // (%xmm0 / x^56) mod x^8
- .elseif \n == 16
- _cond_vex "pextrw $3 + LSB_CRC,", %xmm0, %eax // (%xmm0 / x^48) mod x^16
- .elseif \n == 32
- _cond_vex "pextrd $1 + LSB_CRC,", %xmm0, %eax // (%xmm0 / x^32) mod x^32
- .else // \n == 64 && !LSB_CRC
- _cond_vex movq, %xmm0, %rax // (%xmm0 / x^0) mod x^64
- .endif
-.endif
-
-.if VL > 16
- vzeroupper // Needed when ymm or zmm registers may have been used.
-.endif
-#ifdef __i386__
- pop CONSTS_PTR
-#endif
- RET
-.endm
-
-#define DEFINE_CRC_PCLMUL_FUNCS(prefix, bits, lsb) \
-SYM_FUNC_START(prefix##_pclmul_sse); \
- _crc_pclmul n=bits, lsb_crc=lsb, vl=16, avx_level=0; \
-SYM_FUNC_END(prefix##_pclmul_sse); \
- \
-SYM_FUNC_START(prefix##_vpclmul_avx2); \
- _crc_pclmul n=bits, lsb_crc=lsb, vl=32, avx_level=2; \
-SYM_FUNC_END(prefix##_vpclmul_avx2); \
- \
-SYM_FUNC_START(prefix##_vpclmul_avx512); \
- _crc_pclmul n=bits, lsb_crc=lsb, vl=64, avx_level=512; \
-SYM_FUNC_END(prefix##_vpclmul_avx512);
diff --git a/lib/crc/x86/crc-pclmul-template.h b/lib/crc/x86/crc-pclmul-template.h
index 893119bb7c07..a64cf62ab340 100644
--- a/lib/crc/x86/crc-pclmul-template.h
+++ b/lib/crc/x86/crc-pclmul-template.h
@@ -1,7 +1,7 @@
/* SPDX-License-Identifier: GPL-2.0-or-later */
/*
* Macros for accessing the [V]PCLMULQDQ-based CRC functions that are
- * instantiated by crc-pclmul-template.S
+ * implemented by the crc*-pclmul.c files
*
* Copyright 2025 Google LLC
*
diff --git a/lib/crc/x86/crc16-msb-pclmul.S b/lib/crc/x86/crc16-msb-pclmul.S
deleted file mode 100644
index e9fe248093a8..000000000000
--- a/lib/crc/x86/crc16-msb-pclmul.S
+++ /dev/null
@@ -1,6 +0,0 @@
-/* SPDX-License-Identifier: GPL-2.0-or-later */
-// Copyright 2025 Google LLC
-
-#include "crc-pclmul-template.S"
-
-DEFINE_CRC_PCLMUL_FUNCS(crc16_msb, /* bits= */ 16, /* lsb= */ 0)
diff --git a/lib/crc/x86/crc16-msb-pclmul.c b/lib/crc/x86/crc16-msb-pclmul.c
new file mode 100644
index 000000000000..4526c904263f
--- /dev/null
+++ b/lib/crc/x86/crc16-msb-pclmul.c
@@ -0,0 +1,23 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+/*
+ * CRC-16 (MSB-first, as used by T10 DIF) using [V]PCLMULQDQ, via C intrinsics
+ *
+ * Copyright 2025 Google LLC
+ */
+
+#include <linux/types.h>
+
+#define CRC_BITS 16
+#define CRC_MSB 1
+
+#define VL 16
+#define CRC_FUNC crc16_msb_pclmul_sse
+#include "crc-pclmul-impl.h"
+
+#define VL 32
+#define CRC_FUNC crc16_msb_vpclmul_avx2
+#include "crc-pclmul-impl.h"
+
+#define VL 64
+#define CRC_FUNC crc16_msb_vpclmul_avx512
+#include "crc-pclmul-impl.h"
--
2.56.0
^ permalink raw reply related [flat|nested] 11+ messages in thread* Re: [PATCH RFC v2 0/6] lib/crc: x86: convert the CRC assembly to C intrinsics
2026-11-20 1:18 [PATCH RFC v2 0/6] lib/crc: x86: convert the CRC assembly to C intrinsics Qu Wenruo
` (5 preceding siblings ...)
2026-11-20 1:19 ` [PATCH RFC v2 6/6] lib/crc: x86: Convert crc16_msb_*() " Qu Wenruo
@ 2026-11-20 3:42 ` Eric Biggers
2026-11-20 4:24 ` Qu Wenruo
2026-11-20 8:41 ` Peter Zijlstra
6 siblings, 2 replies; 11+ messages in thread
From: Eric Biggers @ 2026-11-20 3:42 UTC (permalink / raw)
To: Qu Wenruo; +Cc: linux-btrfs, linux-crypto, ardb, x86, linux-kernel
[ +Cc x86@kernel.org ]
Hi Qu,
On Fri, Nov 20, 2026 at 11:48:58AM +1030, Qu Wenruo wrote:
> [CHANGELOG]
> v2:
> - Use compiler's intrinsics header directly
> - Convert all x86 asm files to C
>
> Currently the x86 CRC code is all assembly: the generic [V]PCLMULQDQ
> template (12 functions: crc16, crc32 and crc64, three vector lengths each)
> and crc32c-3way.S. arm and arm64 already use C intrinsics.
>
> This series converts all of it to C with the compiler's intrinsics, and
> removes the assembly. No algorithm, constant or calling convention
> changes, so the glue is untouched.
>
> The goal is maintainability and portability, not speed: C is easier to
> read, review and extend than the macro-heavy assembly.
> And it is also easier to port to userspace tools like btrfs-progs and
> xfsprogs which will require optimized CRC for offline operations.
>
> Tests on a Zen5 VM show no obvious change in performance, except for clang
> with the MSB-first SSE and AVX2 functions, see below.
>
> Patches:
>
> 1: asm/x86-intrinsics.h, a wrapper around <immintrin.h>.
> 2: crc32c_x86_3way().
> 3-6: crc32_lsb, crc64_msb, crc64_lsb and crc16_msb, each with the SSE4.1,
> AVX2 and AVX512BW/VL variants. The template crc-pclmul-impl.h grows
> with them, and the assembly is removed in the last one.
>
> Design decisions and trade-offs
> ===============================
>
> - ISA selection: target attribute instead of per-file flags
>
> Each function and helper has __attribute__((target(...))), the code is
> not built with -msse4.1, -mavx2 or -mavx512*. Per-file flags would
> need one object per ISA level, or the SSE function could use AVX-512.
>
> Pros: all vector lengths of a CRC are in one file, and the compiler can
> only use an extension in the functions that run after the CPU feature
> checks (checked in the objdump).
>
> Cons: no precedent in the kernel for selecting x86 ISA levels this way,
> and this is completely compiler dependent. See the GCC 8.1/8.2 issue
> below.
>
> - One template, included once per function
>
> The 12 functions differ only in vector length, CRC width and bit order,
> so crc-pclmul-impl.h generates one per inclusion, after the includer
> defined VL, CRC_BITS, CRC_MSB and CRC_FUNC.
>
> I tried two different solutions:
>
> * Nine standalone functions
> About 1400 lines against about 490 with the template.
> And any fututre fixes needs to be duplicated 9 times.
>
> * A DEFINE_CRC_PCLMUL_FUNCS() style macro
> It can work, but the body cannot have #if, so the wide-vector stages
> become constant if()s and GCC warns about the dead code (-Wpsabi).
>
> That macro also makes checkpatch warn about flow control in a macro,
> and compiler errors, gdb and perf point at the invocation line.
>
> The include-as-template is not a common practice, but would be the
> least-worst solution I can come up with.
>
> - Generated code and performance
>
> Against the assembly (kbuild objects, GCC 16.2, clang 23): clang has
> the same total size for the 12 pclmul functions (7453 against 7482
> bytes) and the same instruction count in each main loop.
>
> GCC is 18% bigger (SSE +12%, AVX2 +22%, AVX-512 +19%): stack
> realignment in every function and, for AVX-512, loads not folded into
> vpternlogq (19 instead of 16 instructions in the LSB-first main loop).
>
> crc32c_x86_3way() grows from 400 bytes to 603 (GCC) and 783 (clang),
> mostly prologue and unrolling of the tail. The main loop is 15, 17 and
> 16 instructions (asm, GCC, clang).
>
> Every C function also gets endbr64 and the ftrace call, which the
> assembly never had. The call is patched to a nop at boot.
>
> Throughput comes from a userspace harness using the same sources, built
> with the kernel's -mno-sse flags, on a Zen5 VM (Ryzen AI 9 HX 370): one
> pinned CPU, median of 7 runs. No in-kernel benchmark yet.
>
> Peak is about 7.7, 15.4 and 30.7 GB/s for SSE, AVX2 and AVX-512. GCC
> is within 2% of the assembly from 1 KiB, up to 4-6% slower at 64-200
> bytes and 8% for AVX2 at 512 bytes.
>
> clang is the same, except crc64_msb and crc16_msb: SSE is 5-8% slower
> from 512 bytes, AVX2 5-7% from 1 KiB, but 14% faster at 512 bytes.
> AVX-512 is equal.
>
> The clang SSE loop is 175 bytes against 153 for the assembly, as it
> keeps data in xmm8-xmm11 (REX prefixes). That is likely the cause,
> changing the XOR order did not help.
>
> crc32c_x86_3way() is within 3% of the assembly: 8.6 to 9.0 GB/s at 200
> bytes, 14.8 to 14.9 GB/s at 64 KiB.
>
> Known issue: GCC 8.1 and 8.2
>
> GCC 8.1 and 8.2 cannot build the AVX2 functions. Their
> <vpclmulqdqintrin.h> declares _mm256_clmulepi64_epi128() with avx512vl
> in its target, which fold256() lacks, so it cannot be inlined.
>
> GCC 8.3 and later only need avx. I reproduced it on Compiler Explorer,
> where 8.3 to 9.5 build the code cleanly.
>
> Calling __builtin_ia32_vpclmulqdq_v4di() directly for GCC older than 9
> fixes it, but it is not included. 8.1 is still the documented minimum,
> so: add that workaround, raise the minimum, or ignore 8.1 and 8.2?
>
>
> Qu Wenruo (6):
> x86: Add a wrapper header for the compiler's SIMD intrinsics
> lib/crc: x86: Convert crc32c_x86_3way() from assembly to C intrinsics
> lib/crc: x86: Convert crc32_lsb_*() from assembly to C
> lib/crc: x86: Convert crc64_msb_*() from assembly to C
> lib/crc: x86: Convert crc64_lsb_*() from assembly to C
> lib/crc: x86: Convert crc16_msb_*() from assembly to C
>
> arch/x86/include/asm/x86-intrinsics.h | 49 +++
> lib/crc/Makefile | 11 +-
> lib/crc/x86/crc-pclmul-helpers.h | 90 ++++
> lib/crc/x86/crc-pclmul-impl.h | 304 ++++++++++++++
> lib/crc/x86/crc-pclmul-template.S | 575 --------------------------
> lib/crc/x86/crc-pclmul-template.h | 2 +-
> lib/crc/x86/crc16-msb-pclmul.S | 6 -
> lib/crc/x86/crc16-msb-pclmul.c | 23 ++
> lib/crc/x86/crc32-pclmul.S | 6 -
> lib/crc/x86/crc32-pclmul.c | 23 ++
> lib/crc/x86/crc32.h | 2 +-
> lib/crc/x86/crc32c-3way.S | 360 ----------------
> lib/crc/x86/crc32c-3way.c | 264 ++++++++++++
> lib/crc/x86/crc64-lsb-pclmul.c | 23 ++
> lib/crc/x86/crc64-msb-pclmul.c | 23 ++
> lib/crc/x86/crc64-pclmul.S | 7 -
> 16 files changed, 811 insertions(+), 957 deletions(-)
> create mode 100644 arch/x86/include/asm/x86-intrinsics.h
> create mode 100644 lib/crc/x86/crc-pclmul-helpers.h
> create mode 100644 lib/crc/x86/crc-pclmul-impl.h
> delete mode 100644 lib/crc/x86/crc-pclmul-template.S
> delete mode 100644 lib/crc/x86/crc16-msb-pclmul.S
> create mode 100644 lib/crc/x86/crc16-msb-pclmul.c
> delete mode 100644 lib/crc/x86/crc32-pclmul.S
> create mode 100644 lib/crc/x86/crc32-pclmul.c
> delete mode 100644 lib/crc/x86/crc32c-3way.S
> create mode 100644 lib/crc/x86/crc32c-3way.c
> create mode 100644 lib/crc/x86/crc64-lsb-pclmul.c
> create mode 100644 lib/crc/x86/crc64-msb-pclmul.c
> delete mode 100644 lib/crc/x86/crc64-pclmul.S
Thanks for exploring this.
I think we should be honest: this would be a negative for
maintainability at least in the short term. We'd be replacing existing
well-optimized and well-documented code with an experimental use of x86
C intrinsics in the kernel, with many of the original comments stripped
out or made inconsistent, and signing up to fight with compiler issues
(including inconsistent support between compilers, having to use
error-prone hacks to use the x86 intrinsics in the kernel at all, and
inefficient code generation which in some cases might not be solvable
without contributing fixes to gcc/clang).
I've used the x86 C intrinsics extensively in userspace projects. So
sure, some of these compiler issues (minus the ones specific to trying
to use the intrinsics in the kernel) are things I'm used to already.
It's just been nice to not have to deal with them in the kernel.
Note that gcc can't even generate simple loops using vector intrinsics
without emitting unnecessary move instructions. I reported it years ago
(https://gcc.gnu.org/bugzilla/show_bug.cgi?id=107892) and it's still
unfixed. So realistically, we either need to simply tolerate bad
codegen like that, or actually step up and actively contribute fixes to
the compilers (as merely opening a bug report does not result in a fix).
And as you mentioned in your cover letter, a lot of the code grew in
size and became slower, especially for short inputs which are important
for some in-kernel use cases. So clearly there is some bad codegen
going on compared to what was there before.
I kind of feel this could only be a benefit in the long term if we
actually go all in and commit to switching the crypto code over to this
style of implementation as well. But that area has a lot more code.
And that code also has another thorny issue: when operating on secrets,
it has to make sure to only use x86 instructions that are in the DOITM
(Data Operand Independent Timing Mode) subset. Yet there's no compiler
support for restricting the used instructions to that subset, and no
announced plans from anyone (Intel, AMD, or otherwise) to implement it.
I'll also note that the hardest part to understand about this code (IMO)
is actually the polynomial math, especially the Barrett reduction. And
yet this series deletes or greatly shortens many of the comments
explaining it, leaving unexplained operations, factually wrong comments
(especially for the Barrett reduction), and dangling references. That's
most of the negative diff, too.
So if we're going to proceed, I'd at least like to see a version that
compares a bit more favorably to the original. But also I think before
diving into this approach I'd like wider agreement in the Linux
community (x86 maintainers, crypto, etc.) that the x86 intrinsics are
the way to go, and for there to be a path forward for DOITM as well, and
hopefully some interest in improving the compilers too.
- Eric
^ permalink raw reply [flat|nested] 11+ messages in thread* Re: [PATCH RFC v2 0/6] lib/crc: x86: convert the CRC assembly to C intrinsics
2026-11-20 3:42 ` [PATCH RFC v2 0/6] lib/crc: x86: convert the CRC assembly to C intrinsics Eric Biggers
@ 2026-11-20 4:24 ` Qu Wenruo
2026-11-20 8:41 ` Peter Zijlstra
1 sibling, 0 replies; 11+ messages in thread
From: Qu Wenruo @ 2026-11-20 4:24 UTC (permalink / raw)
To: Eric Biggers, Qu Wenruo
Cc: linux-btrfs, linux-crypto, ardb, x86, linux-kernel
在 2026/11/20 14:12, Eric Biggers 写道:
> [ +Cc x86@kernel.org ]
>
> Hi Qu,
>
> On Fri, Nov 20, 2026 at 11:48:58AM +1030, Qu Wenruo wrote:
[...]
>
> Thanks for exploring this.
>
> I think we should be honest: this would be a negative for
> maintainability at least in the short term. We'd be replacing existing
> well-optimized and well-documented code with an experimental use of x86
> C intrinsics in the kernel, with many of the original comments stripped
> out or made inconsistent,
That's indeed a huge problem, thanks for pointing this out.
Sorry that I didn't manually compare the comment part before and after
the change, that's all my bad.
That's in fact the most important part for readability, and my focus on
the generated code blinded me on this important part.
> and signing up to fight with compiler issues
> (including inconsistent support between compilers, having to use
> error-prone hacks to use the x86 intrinsics in the kernel at all,
To be honest, to make kernel use intrinsics with current latest
gcc/clang is easier than my initial expectation.
And even for ancient ones, only GCC 8.1/8.2 are really problematic,
again better than my expectation, but still requires extra adhoc fixes
for certain compilers.
> and
> inefficient code generation which in some cases might not be solvable
> without contributing fixes to gcc/clang).
>
> I've used the x86 C intrinsics extensively in userspace projects. So
> sure, some of these compiler issues (minus the ones specific to trying
> to use the intrinsics in the kernel) are things I'm used to already.
> It's just been nice to not have to deal with them in the kernel.
>
> Note that gcc can't even generate simple loops using vector intrinsics
> without emitting unnecessary move instructions. I reported it years ago
> (https://gcc.gnu.org/bugzilla/show_bug.cgi?id=107892) and it's still
> unfixed. So realistically, we either need to simply tolerate bad
> codegen like that, or actually step up and actively contribute fixes to
> the compilers (as merely opening a bug report does not result in a fix).
Personally speaking I'd also prefer the compiler to do a lot of things
better, include not pulling in the memory allocation headers just for
x86 intrinsics.
>
> And as you mentioned in your cover letter, a lot of the code grew in
> size and became slower, especially for short inputs which are important
> for some in-kernel use cases. So clearly there is some bad codegen
> going on compared to what was there before.
>
> I kind of feel this could only be a benefit in the long term if we
> actually go all in and commit to switching the crypto code over to this
> style of implementation as well. But that area has a lot more code.
> And that code also has another thorny issue: when operating on secrets,
> it has to make sure to only use x86 instructions that are in the DOITM
> (Data Operand Independent Timing Mode) subset. Yet there's no compiler
> support for restricting the used instructions to that subset, and no
> announced plans from anyone (Intel, AMD, or otherwise) to implement it.
>
> I'll also note that the hardest part to understand about this code (IMO)
> is actually the polynomial math, especially the Barrett reduction. And
> yet this series deletes or greatly shortens many of the comments
> explaining it, leaving unexplained operations, factually wrong comments
> (especially for the Barrett reduction), and dangling references. That's
> most of the negative diff, too.
>
> So if we're going to proceed, I'd at least like to see a version that
> compares a bit more favorably to the original. But also I think before
> diving into this approach I'd like wider agreement in the Linux
> community (x86 maintainers, crypto, etc.) that the x86 intrinsics are
> the way to go, and for there to be a path forward for DOITM as well, and
> hopefully some interest in improving the compilers too.
Sure, that's a very reasonable argument, and I'm definitely stepping
beyond my ability and initial objective (just to use the same intrinsics
code in both kernel and progs).
I can definitely wait for more feedback on this topic before continue
any future work.
And to my initial objective, I'd check other user space CRC32C
implementation as a reference for now.
Thanks,
Qu
>
> - Eric
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH RFC v2 0/6] lib/crc: x86: convert the CRC assembly to C intrinsics
2026-11-20 3:42 ` [PATCH RFC v2 0/6] lib/crc: x86: convert the CRC assembly to C intrinsics Eric Biggers
2026-11-20 4:24 ` Qu Wenruo
@ 2026-11-20 8:41 ` Peter Zijlstra
2026-11-20 14:12 ` David Laight
1 sibling, 1 reply; 11+ messages in thread
From: Peter Zijlstra @ 2026-11-20 8:41 UTC (permalink / raw)
To: Eric Biggers
Cc: Qu Wenruo, linux-btrfs, linux-crypto, ardb, x86, linux-kernel
On Thu, Nov 19, 2026 at 07:42:20PM -0800, Eric Biggers wrote:
> So if we're going to proceed, I'd at least like to see a version that
> compares a bit more favorably to the original. But also I think before
> diving into this approach I'd like wider agreement in the Linux
> community (x86 maintainers, crypto, etc.) that the x86 intrinsics are
> the way to go, and for there to be a path forward for DOITM as well, and
> hopefully some interest in improving the compilers too.
My 2cents go towards keeping the asm. There is no point in fighting
compilers over this. The very same is true for atomics.
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH RFC v2 0/6] lib/crc: x86: convert the CRC assembly to C intrinsics
2026-11-20 8:41 ` Peter Zijlstra
@ 2026-11-20 14:12 ` David Laight
0 siblings, 0 replies; 11+ messages in thread
From: David Laight @ 2026-11-20 14:12 UTC (permalink / raw)
To: Peter Zijlstra
Cc: Eric Biggers, Qu Wenruo, linux-btrfs, linux-crypto, ardb, x86,
linux-kernel
On Fri, 20 Nov 2026 09:41:58 +0100
Peter Zijlstra <peterz@infradead.org> wrote:
> On Thu, Nov 19, 2026 at 07:42:20PM -0800, Eric Biggers wrote:
>
> > So if we're going to proceed, I'd at least like to see a version that
> > compares a bit more favorably to the original. But also I think before
> > diving into this approach I'd like wider agreement in the Linux
> > community (x86 maintainers, crypto, etc.) that the x86 intrinsics are
> > the way to go, and for there to be a path forward for DOITM as well, and
> > hopefully some interest in improving the compilers too.
>
> My 2cents go towards keeping the asm. There is no point in fighting
> compilers over this. The very same is true for atomics.
>
Indeed.
IIRC the code loop had to go asm block because the compiler pretty much
refused to generate the loop that matched the source, instead it
generated one that added extra clock(s) to the loop making it slower.
I've also use some of the 'intrinsics' to write a short C function.
I basically worked out the asm instructions I wanted then tried to find
the correct intrinsic.
The whole thing would have been just as readable in asm
(possibly with some #defines to give the registers names).
David
^ permalink raw reply [flat|nested] 11+ messages in thread