RE: [PATCH] eal/x86: optimize memcpy of small 64-byte blocks

["Morten Brørup" <mb@smartsharesystems.com>]

> From: Morten Brørup
> Sent: Tuesday, 18 August 2026 11.13
> 
> > From: Konstantin Ananyev [mailto:konstantin.ananyev@huawei.com]
> > Sent: Tuesday, 18 August 2026 10.15
> >
> > > > > > > > > > > > > > > +	/* Common way for small copy size of 64-
> > byte
> > > > > > blocks.
> > > > > > > > > > > > Unlikely, so
> > > > > > > > > > > > > > constant size only */
> > > > > > > > > > > > > > > +	if (__rte_constant(n) && (n & 63) == 0 &&
> > n <=
> > > > > > > > > > > > > > RTE_MEMCPY_BLOCK_64_MAX) {
> > > > > > > > > > > > > > > +		void *ret = dst;
> > > > > > > > > > > > > > > +
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Maybe just let compiler decide, it will
> > generate
> > > > vector
> > > > > > > > > > > > instructions in
> > > > > > > > > > > > > > most cases.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 	if (__rte_constant(n))
> > > > > > > > > > > > > > 		return mempcpy(dst, src, n);
> > > > > > > > > > > > >
> > > > > > > > > > > > > Maybe in most, but not in all:
> > > > > > > > > > > > > https://godbolt.org/z/KvdKqT5rY
> > > > > > > > > > > >
> > > > > > > > > > > > With '-mavx' or '-mavx512f' it looks like it does
> > for
> > > > your
> > > > > > > > sample
> > > > > > > > > > code.
> > > > > > > > > > >
> > > > > > > > > > > It also does with -msse4.2 when SZ is reduced to
> 256
> > > > bytes.
> > > > > > > > > > > Clang switches to inline when SZ is reduced to 128
> > bytes.
> > > > > > > > > > >
> > > > > > > > > > > It seems the compiler has a threshold for when to
> > inline
> > > > and
> > > > > > when
> > > > > > > > to
> > > > > > > > > > call the C
> > > > > > > > > > > library's memcpy subroutine.
> > > > > > > > > > > The threshold depends on both copy size and vector
> > > > register
> > > > > > size.
> > > > > > > > > > > And it is compiler dependent.
> > > > > > > > > >
> > > > > > > > > > I think there are compiler options to specify desired
> > > > threshold
> > > > > > > > values.
> > > > > > > > > > Let say for gcc there is  ' -mmemcpy-
> > strategy=strategy'.
> > > > > > > > > > For that example in that particular case
> > > > > > > > > > -mmemcpy-strategy=vector_loop:512:align,loop:-1:align
> > > > > > > > > > generates sse loads/stores.
> > > > > > > > > > Might be we can exploit it somehow?
> > > > > > > > >
> > > > > > > > > That could give us higher granularity/control over
> memcpy
> > for
> > > > > > > > individual
> > > > > > > > > memcpy instances; might be useful for hot code paths
> > where we
> > > > > > have
> > > > > > > > more
> > > > > > > > > knowledge about the copy operation than the compiler
> can
> > > > infer.
> > > > > > > > > However, pragmas are discouraged in DPDK, and this
> looks
> > like
> > > > a
> > > > > > very
> > > > > > > > similar
> > > > > > > > > path.
> > > > > > > >
> > > > > > > > Well, right now  rte_memcpy.h is 700+ lines and keeps
> > growing.
> > > > > > > > Considering that probably pragmas are not that bad.
> > > > > > > > Of course, pragmas have their own issues and it is hard
> to
> > > > ensure
> > > > > > that
> > > > > > > > they will produce same code between different
> > > > compilers/versions,
> > > > > > etc.
> > > > > > > >
> > > > > > > > > > I am not really happy that our home-brewed memcpy
> code-
> > > > block
> > > > > > keeps
> > > > > > > > > > growing,
> > > > > > > > > > while we keep talking that it would be good to
> > eliminate it
> > > > > > > > completely.
> > > > > > > > >
> > > > > > > > > I agree in principle.
> > > > > > > > > However, this rte_memcpy() optimization is for the
> > > > pile/mempool
> > > > > > > > optimizations
> > > > > > > > > I'm working on, so there is a specific use case
> > motivating
> > > > the
> > > > > > added
> > > > > > > > code.
> > > > > > > >
> > > > > > > > I understand that you probably have some specific use-
> case
> > in
> > > > mind.
> > > > > > > > BTW for this optimization you mentioned above: what is
> the
> > gain
> > > > > > with
> > > > > > > > these changes?
> > > > > > >
> > > > > > > IMO, the primary benefit is the much simpler (and smaller)
> > > > assembly
> > > > > > output due
> > > > > > > to avoiding the address alignment check (and the resulting
> > > > duplicated
> > > > > > code).
> > > > > >
> > > > > > I think that should be measurable too: whole binary  and/or
> hot
> > > > path
> > > > > > function size reduction, etc.
> > > > >
> > > > > The size of the generated code for the copy operation is
> reduced
> > to
> > > > slightly less
> > > > > than half (only one instance of the copy operation instead of
> > two,
> > > > and the
> > > > > address alignment comparison is omitted).
> > > > >
> > > > > >
> > > > > > > I haven't measured the performance gain.
> > > > > > > Based on the perf gain in a previous mempool optimization
> > patch
> > > > [1],
> > > > > > it seems
> > > > > > > avoiding the address alignment check shaves ~2 cycles off
> the
> > > > copy
> > > > > > operation (for
> > > > > > > cache-to-cache copy).
> > > > > > > I expect that the same gain (from avoiding the address
> > alignment
> > > > > > check) applies
> > > > > > > here.
> > > > > >
> > > > > > Ok, then I suggest we do some measurements first, before
> going
> > > > forward
> > > > > > with it.
> > > > >
> > > > > I tried memcpy_perf_autotest, but the branch predictor kicks in
> > and
> > > > eliminates
> > > > > the cost of the address alignment comparison. So the results
> are
> > very
> > > > similar.
> > > >
> > > > Indeed, they are the same.
> > > >
> > > > > BTW, that's a general issue with our perf tests: Repeated
> testing
> > > > doesn't show
> > > > > the cost of branches, because they are always eliminated by the
> > > > branch
> > > > > predictor.
> > > >
> > > > Might be...
> > > > But we do need some measurable evidence that such change improve
> > > > things,
> > > > otherwise - why to bother?
> > > > If that perf test is not goof enough - let's try to extend it or
> > use
> > > > different one.
> > > > BTW, in your mempool pile RFC, I see you also use rte_memcpy over
> > fixed
> > > > sized buffers.
> > > > Do you see any improvement there (wit/without rte_memcpy
> > optimixation)?
> > >
> > > Looking at the generated assembly (objdump -S
> > > build/drivers/librte_mempool_stack.so)...
> > >
> > > This is the optimized copy loop in pile_dequeue:
> > >
> > >     1650:	89 ca                	mov    %ecx,%edx
> > >     1652:	c5 fe 6f 40 40       	vmovdqu 0x40(%rax),%ymm0
> > >     1657:	ff c1                	inc    %ecx
> > >     1659:	c1 e2 05             	shl    $0x5,%edx
> > >     165c:	49 8d 14 d4          	lea    (%r12,%rdx,8),%rdx
> > >     1660:	c5 fe 7f 02          	vmovdqu %ymm0,(%rdx)
> > >     1664:	c5 fe 6f 48 60       	vmovdqu 0x60(%rax),%ymm1
> > >     1669:	c5 fe 7f 4a 20       	vmovdqu %ymm1,0x20(%rdx)
> > >     166e:	c5 fe 6f 90 80 00 00 	vmovdqu 0x80(%rax),%ymm2
> > >     1675:	00
> > >     1676:	c5 fe 7f 52 40       	vmovdqu %ymm2,0x40(%rdx)
> > >     167b:	c5 fe 6f 98 a0 00 00 	vmovdqu 0xa0(%rax),%ymm3
> > >     1682:	00
> > >     1683:	c5 fe 7f 5a 60       	vmovdqu %ymm3,0x60(%rdx)
> > >     1688:	c5 fe 6f a0 c0 00 00 	vmovdqu 0xc0(%rax),%ymm4
> > >     168f:	00
> > >     1690:	c5 fe 7f a2 80 00 00 	vmovdqu %ymm4,0x80(%rdx)
> > >     1697:	00
> > >     1698:	c5 fe 6f a8 e0 00 00 	vmovdqu 0xe0(%rax),%ymm5
> > >     169f:	00
> > >     16a0:	c5 fe 7f aa a0 00 00 	vmovdqu %ymm5,0xa0(%rdx)
> > >     16a7:	00
> > >     16a8:	c5 fe 6f b0 00 01 00 	vmovdqu 0x100(%rax),%ymm6
> > >     16af:	00
> > >     16b0:	c5 fe 7f b2 c0 00 00 	vmovdqu %ymm6,0xc0(%rdx)
> > >     16b7:	00
> > >     16b8:	c5 fe 6f b8 20 01 00 	vmovdqu 0x120(%rax),%ymm7
> > >     16bf:	00
> > >     16c0:	c5 fe 7f ba e0 00 00 	vmovdqu %ymm7,0xe0(%rdx)
> > >     16c7:	00
> > >     16c8:	48 8b 40 08          	mov    0x8(%rax),%rax
> > >     16cc:	39 f1                	cmp    %esi,%ecx
> > >     16ce:	75 80                	jne    1650
> <pile_dequeue+0xe0>
> > >     16d0:
> > >
> > > This is without the rte_memcpy optimization:
> > >
> > >     1650:	89 ca                	mov    %ecx,%edx
> > >     1652:	c5 fe 6f 40 40       	vmovdqu 0x40(%rax),%ymm0
> > >     1657:	49 89 c1             	mov    %rax,%r9
> > >     165a:	ff c1                	inc    %ecx
> > >     165c:	c1 e2 05             	shl    $0x5,%edx
> > >     165f:	49 8d 14 d4          	lea    (%r12,%rdx,8),%rdx
> > >     1663:	c5 fe 7f 02          	vmovdqu %ymm0,(%rdx)
> > >     1667:	c5 fe 6f 48 60       	vmovdqu 0x60(%rax),%ymm1
> > >     166c:	49 09 d1             	or     %rdx,%r9
> > >     166f:	41 83 e1 1f          	and    $0x1f,%r9d
> > >     1673:	c5 fe 7f 4a 20       	vmovdqu %ymm1,0x20(%rdx)
> > >     1678:	c5 fe 6f 90 80 00 00 	vmovdqu 0x80(%rax),%ymm2
> > >     167f:	00
> > >     1680:	c5 fe 7f 52 40       	vmovdqu %ymm2,0x40(%rdx)
> > >     1685:	c5 fe 6f 98 a0 00 00 	vmovdqu 0xa0(%rax),%ymm3
> > >     168c:	00
> > >     168d:	c5 fe 7f 5a 60       	vmovdqu %ymm3,0x60(%rdx)
> > >     1692:	c5 fe 6f a0 c0 00 00 	vmovdqu 0xc0(%rax),%ymm4
> > >     1699:	00
> > >     169a:	c5 fe 7f a2 80 00 00 	vmovdqu %ymm4,0x80(%rdx)
> > >     16a1:	00
> > >     16a2:	c5 fe 6f a8 e0 00 00 	vmovdqu 0xe0(%rax),%ymm5
> > >     16a9:	00
> > >     16aa:	c5 fe 7f aa a0 00 00 	vmovdqu %ymm5,0xa0(%rdx)
> > >     16b1:	00
> > >     16b2:	c5 fe 6f b0 00 01 00 	vmovdqu 0x100(%rax),%ymm6
> > >     16b9:	00
> > >     16ba:	c5 fe 7f b2 c0 00 00 	vmovdqu %ymm6,0xc0(%rdx)
> > >     16c1:	00
> > >     16c2:	c5 fe 6f b8 20 01 00 	vmovdqu 0x120(%rax),%ymm7
> > >     16c9:	00
> > >     16ca:	c5 fe 7f ba e0 00 00 	vmovdqu %ymm7,0xe0(%rdx)
> > >     16d1:	00
> > >     16d2:	48 8b 40 08          	mov    0x8(%rax),%rax
> > >     16d6:	0f 85 c4 00 00 00    	jne    17a0
> <pile_dequeue+0x230>
> > >     16dc:	39 ce                	cmp    %ecx,%esi
> > >     16de:	0f 85 6c ff ff ff    	jne    1650
> <pile_dequeue+0xe0>
> > >     16e4:
> > >     [...]
> > >     17a0:	39 f1                	cmp    %esi,%ecx
> > >     17a2:	0f 85 a8 fe ff ff    	jne    1650
> <pile_dequeue+0xe0>
> > >     17a8:	e9 37 ff ff ff       	jmp    16e4
> <pile_dequeue+0x174>
> > >
> > > Here, the alignment comparison I have optimized away is performed
> > using
> > > register %r9.
> > >
> > > With the optimization, the generated assembly is less cluttered,
> and
> > thus easier
> > > to review.
> > > (It seems the compiler is clever enough to not duplicate the two
> > instances of the
> > > code, but reuse the aligned instance for the unaligned case too. So
> > the reduction
> > > in code size is not as great as I previously claimed.)
> > >
> > > I agree the performance benefit in CPU cycles is probably
> > insignificant.
> > > But the generated assembly is cleaner.
> > > And if pressured for CPU registers, the optimization also frees up
> > one CPU
> > > register for other purposes.
> > >
> > > Code size reduction:
> > > The optimized loop is 0x80 bytes of instructions.
> > > The non-optimized is 0x8e bytes, 14 bytes more, in the loop, plus
> 13
> > bytes
> > > outside the loop.
> >
> > Honestly, with such insignificant gains, I'd either leave it alone,
> > or look more closely at compiler pragmas option we discussed before.
> 
> Compiled code is slightly smaller, performance is slightly better.
> But source code having slightly more lines of code is more important?
> 
> If the source code added was difficult to read, complexity was
> increased, or had side effects or spillover to other modules, I might
> buy that argument.
> But the addition is very simple and completely isolated.
> 
> If the objection is about source code readability, I could add more
> inline comments to the added code, elaborating that the added code path
> is the same for all CPU vector sizes, regardless of address alignment.
> But I suspect such comments would confuse more than they would help.
> And it would add even more lines to the file size.
> 
> Adding compiler pragmas should probably be done in source code calling
> rte_memcpy(), like __rte_assume() and the coming
> __rte_assume_cache_aligned() [1].
> 
> [1]:
> https://patchwork.dpdk.org/project/dpdk/patch/20260812090723.1771628-1-
> mb@smartsharesystems.com/
> 
> If you try experimenting with compiler pragmas, I'm open for reviewing
> an RFC!
> For now, let's take this small step.
> 

I seems nobody want more code in the rte_memcpy() implementation, so I have marked this patch as Rejected, and instead posted a separate patch with the cleanups [2].

[2]: https://patchwork.dpdk.org/project/dpdk/patch/20260904092028.856816-1-mb@smartsharesystems.com/

I had an improved version in the works, not requiring constant size; but I assume it will be rejected too, so I'm not pursuing it.
For anyone interested, it had this criteria:

	if (__rte_constant((n & 63) == 0 && n <= RTE_MEMCPY_BLOCK_64_MAX) &&
		 (n & 63) == 0 && n <= RTE_MEMCPY_BLOCK_64_MAX) {

The full diff:

@@ -175,2 +175,3 @@ rte_mov256(uint8_t *__rte_restrict dst,
 #define RTE_MEMCPY_ALIGNMENT_MASK 0x3F
+#define RTE_MEMCPY_BLOCK_64_MAX 512
 
@@ -321,2 +322,3 @@ COPY_BLOCK_128_BACK63:
 #define RTE_MEMCPY_ALIGNMENT_MASK 0x1F
+#define RTE_MEMCPY_BLOCK_64_MAX 256
 
@@ -432,2 +434,3 @@ COPY_BLOCK_128_BACK31:
 #define RTE_MEMCPY_ALIGNMENT_MASK 0x0F
+#define RTE_MEMCPY_BLOCK_64_MAX 512
 
@@ -675,2 +678,36 @@ rte_memcpy(void *__rte_restrict dst, con
 {
+	/*
+	 * Common way for small copy size of 64-byte blocks,
+	 * independent of alignment with vector register size.
+	 * Note:
+	 * If known at compile time that the criteria are fulfilled,
+	 * the compiler will use this, and omit everything else;
+	 * otherwise, the compiler will omit this.
+	 */
+	static_assert(RTE_MEMCPY_BLOCK_64_MAX <= 512,
+			"64-byte block copy max size too big for implementation below");
+	if (__rte_constant((n & 63) == 0 && n <= RTE_MEMCPY_BLOCK_64_MAX) &&
+			(n & 63) == 0 && n <= RTE_MEMCPY_BLOCK_64_MAX) {
+		void *ret = dst;
+
+		if (!(RTE_MEMCPY_BLOCK_64_MAX < 512) && n == 512) {
+			rte_mov256((uint8_t *)dst + 0 * 256, (const uint8_t *)src + 0 * 256);
+			rte_mov256((uint8_t *)dst + 1 * 256, (const uint8_t *)src + 1 * 256);
+		} else {
+			if (n & 256) {
+				rte_mov256((uint8_t *)dst, (const uint8_t *)src);
+				src = (const uint8_t *)src + 256;
+				dst = (uint8_t *)dst + 256;
+			}
+			if (n & 128) {
+				rte_mov128((uint8_t *)dst, (const uint8_t *)src);
+				src = (const uint8_t *)src + 128;
+				dst = (uint8_t *)dst + 128;
+			}
+			if (n & 64)
+				rte_mov64((uint8_t *)dst, (const uint8_t *)src);
+		}
+		return ret;
+	}
+
 	/* Fast way when copy size doesn't exceed 64 bytes. */
@@ -715,2 +752,3 @@ rte_memcpy(void *__rte_restrict dst, con
 #undef RTE_MEMCPY_ALIGNMENT_MASK
+#undef RTE_MEMCPY_BLOCK_64_MAX