Re: [PATCH v2] powerpc/lib: prefetch ahead in __csum_partial()

["Christophe Leroy (CS GROUP)" <chleroy@kernel.org>]

Le 12/10/2026 à 20:00, Rosen Penev a écrit :
> __csum_partial() walks the buffer linearly and reads each cache line
> exactly once. On cores without a hardware prefetcher, such as the 464FP
> in the APM82181, every new line is therefore a demand miss that stalls
> the loop. Packets received by non-coherent DMA are never in the cache,
> so this is the common case for GRO checksum validation: when routing
> TCP with GRO enabled on a Meraki MX60, __csum_partial() takes about 30%
> of all cycles and is the top entry in the profile.
> 
> Issue a dcbt four cache lines (128 bytes) ahead in the main loop.
> dcbt is only a hint and never faults, so prefetching past the end of
> the buffer is harmless. Lines that are already cached, for example when
> checksumming data the CPU just wrote, are left untouched.
> 
> Measured on a Meraki MX60 (APM82181 at 800 MHz), single TCP stream
> routed through a qca8k DSA switch, iperf3 median of 5. Only the dcbt
> was toggled at runtime, in the same boot:
> 
>                LAN->WAN               WAN->LAN
>                Mbit/s  csum cycles    Mbit/s  csum cycles
>    before      495     30.5%          439     27.2%
>    after       599     15.3%          520     13.7%
> 
> That is 58% less time spent in __csum_partial() per byte.
> 
> The prefetch distance was chosen with an earlier, slower driver
> configuration (median of 3, Mbit/s):
> 
>    distance    LAN->WAN   WAN->LAN
>    none        370        341
>    64 bytes    430        381
>    96 bytes    438        397
>    128 bytes   438        385
>    160 bytes   434        386
>    256 bytes   423        383
> 
> Prefetching the first lines before entering the loop made no
> measurable difference and is not done.
> 
> Assisted-by: LLM
> Signed-off-by: Rosen Penev <rosenp@gmail.com>

csum_partial_copy_generic() does the same, so your patch makes sense.

Wondering if you should use MAX_COPY_PREFETCH instead of hardcoding value 4.

Reviewed-by: Christophe Leroy (CS GROUP) <chleroy@kernel.org>

> ---
>   v2: reword commit message and retest numbers.
>   arch/powerpc/lib/checksum_32.S | 4 +++-
>   1 file changed, 3 insertions(+), 1 deletion(-)
> 
> diff --git a/arch/powerpc/lib/checksum_32.S b/arch/powerpc/lib/checksum_32.S
> index cd00b9bdd772..5a14d45c7b9d 100644
> --- a/arch/powerpc/lib/checksum_32.S
> +++ b/arch/powerpc/lib/checksum_32.S
> @@ -43,6 +43,7 @@ _GLOBAL(__csum_partial)
>   	bdnz	2b
>   21:	srwi.	r6,r4,4		/* # blocks of 4 words to do */
>   	beq	3f
> +	li	r9,4*L1_CACHE_BYTES	/* prefetch distance */
>   	lwz	r0,4(r3)
>   	mtctr	r6
>   	lwz	r6,8(r3)
> @@ -52,7 +53,8 @@ _GLOBAL(__csum_partial)
>   	lwzu	r8,16(r3)
>   	adde	r5,r5,r7
>   	bdz	23f
> -22:	lwz	r0,4(r3)
> +22:	dcbt	r3,r9
> +	lwz	r0,4(r3)
>   	adde	r5,r5,r8
>   	lwz	r6,8(r3)
>   	adde	r5,r5,r0