[PATCH] MIPS: lib: prefetch ahead in csum_partial()

[PATCH] MIPS: lib: prefetch ahead in csum_partial()

[Rosen Penev]

csum_partial() reads every byte of the buffer once. Cores like the 74K
have no hardware prefetcher, so every cache line is a demand miss, and
after a non-coherent DMA the received data is never in the cache. With
GRO enabled, validating the checksum of forwarded TCP traffic made
csum_partial() the top entry in the profile on a BCM4716 (74Kc at
480 MHz, no L2, ~80 cycles per DRAM miss): 13-16% of all cycles.

Prefetch the 128 byte block two iterations ahead in the main loop.
Unlike memcpy(), which disables prefetching on DMA_NONCOHERENT
entirely, only prefetch blocks that lie inside the buffer: the 74K does
not need a post-DMA invalidate, so a line prefetched past the end could
belong to a buffer the device is still writing, and the CPU would later
read the stale copy.

On an Asus RT-N16 (BCM4716, bgmac, single TCP stream unless noted,
iperf3 median of 3, A/B in the same boot by toggling the prefetch,
two rounds each):

                       without      with      (Mbit/s)
  local rx / tx        220 / 194    243 / 212
  routed up / down     169 / 140    187 / 154
  routed, 4 streams    65 / 79      69 / 84
  flow offload bidir   253          292

Assisted-by: LLM
Signed-off-by: Rosen Penev <rosenp@gmail.com>
---
 arch/mips/lib/csum_partial.S | 15 +++++++++++++++
 1 file changed, 15 insertions(+)

diff --git a/arch/mips/lib/csum_partial.S b/arch/mips/lib/csum_partial.S
index 3d2ff4118d79..f0ec47259ef4 100644
--- a/arch/mips/lib/csum_partial.S
+++ b/arch/mips/lib/csum_partial.S
@@ -190,6 +190,21 @@ EXPORT_SYMBOL(csum_partial)
 	 andi	t2, a1, 0x40
 
 .Lmove_128bytes:
+#ifdef CONFIG_CPU_HAS_PREFETCH
+	/*
+	 * Fetch the block two iterations ahead, but only while it lies
+	 * inside the buffer: on non-coherent systems a line prefetched
+	 * past the end could belong to a buffer the device still owns.
+	 */
+	sltiu	t5, t8, 3
+	bnez	t5, 2f
+	 nop
+	pref	0, 0x100(src)
+	pref	0, 0x120(src)
+	pref	0, 0x140(src)
+	pref	0, 0x160(src)
+2:
+#endif
 	CSUM_BIGCHUNK(src, 0x00, sum, t0, t1, t3, t4)
 	CSUM_BIGCHUNK(src, 0x20, sum, t0, t1, t3, t4)
 	CSUM_BIGCHUNK(src, 0x40, sum, t0, t1, t3, t4)
-- 
2.56.0

Re: [PATCH] MIPS: lib: prefetch ahead in csum_partial()

[kernel test robot]

Hi Rosen,

kernel test robot noticed the following build errors:

[auto build test ERROR on linus/master]
[also build test ERROR on v7.3-rc6 next-20261009]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/Rosen-Penev/MIPS-lib-prefetch-ahead-in-csum_partial/20261009-235442
base:   linus/master
patch link:    https://lore.kernel.org/r/20261010065442.23780-1-rosenp%40gmail.com
patch subject: [PATCH] MIPS: lib: prefetch ahead in csum_partial()
config: mips-rbtx49xx_defconfig (https://download.01.org/0day-ci/archive/20261013/202610131201.wWk11MLy-lkp@intel.com/config)
compiler: mips-linux-gcc (GCC) 16.1.0
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20261013/202610131201.wWk11MLy-lkp@intel.com/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <lkp@intel.com>
| Closes: https://lore.kernel.org/oe-kbuild-all/202610131201.wWk11MLy-lkp@intel.com/

All errors (new ones prefixed by >>):

   Assembler messages:
>> arch/mips/lib/csum_partial.S:202: Error: opcode not supported on this processor: r4600 (mips3) `pref 0,0x100($4)'
   arch/mips/lib/csum_partial.S:203: Error: opcode not supported on this processor: r4600 (mips3) `pref 0,0x120($4)'
   arch/mips/lib/csum_partial.S:204: Error: opcode not supported on this processor: r4600 (mips3) `pref 0,0x140($4)'
   arch/mips/lib/csum_partial.S:205: Error: opcode not supported on this processor: r4600 (mips3) `pref 0,0x160($4)'


vim +202 arch/mips/lib/csum_partial.S

   102	
   103		.text
   104		.set	noreorder
   105		.align	5
   106	LEAF(csum_partial)
   107	EXPORT_SYMBOL(csum_partial)
   108		move	sum, zero
   109		move	t7, zero
   110	
   111		sltiu	t8, a1, 0x8
   112		bnez	t8, .Lsmall_csumcpy		/* < 8 bytes to copy */
   113		 move	t2, a1
   114	
   115		andi	t7, src, 0x1			/* odd buffer? */
   116	
   117	.Lhword_align:
   118		beqz	t7, .Lword_align
   119		 andi	t8, src, 0x2
   120	
   121		lbu	t0, (src)
   122		LONG_SUBU	a1, a1, 0x1
   123	#ifdef __MIPSEL__
   124		sll	t0, t0, 8
   125	#endif
   126		ADDC(sum, t0)
   127		PTR_ADDU	src, src, 0x1
   128		andi	t8, src, 0x2
   129	
   130	.Lword_align:
   131		beqz	t8, .Ldword_align
   132		 sltiu	t8, a1, 56
   133	
   134		lhu	t0, (src)
   135		LONG_SUBU	a1, a1, 0x2
   136		ADDC(sum, t0)
   137		sltiu	t8, a1, 56
   138		PTR_ADDU	src, src, 0x2
   139	
   140	.Ldword_align:
   141		bnez	t8, .Ldo_end_words
   142		 move	t8, a1
   143	
   144		andi	t8, src, 0x4
   145		beqz	t8, .Lqword_align
   146		 andi	t8, src, 0x8
   147	
   148		LOAD32	t0, 0x00(src)
   149		LONG_SUBU	a1, a1, 0x4
   150		ADDC(sum, t0)
   151		PTR_ADDU	src, src, 0x4
   152		andi	t8, src, 0x8
   153	
   154	.Lqword_align:
   155		beqz	t8, .Loword_align
   156		 andi	t8, src, 0x10
   157	
   158	#ifdef USE_DOUBLE
   159		ld	t0, 0x00(src)
   160		LONG_SUBU	a1, a1, 0x8
   161		ADDC(sum, t0)
   162	#else
   163		lw	t0, 0x00(src)
   164		lw	t1, 0x04(src)
   165		LONG_SUBU	a1, a1, 0x8
   166		ADDC(sum, t0)
   167		ADDC(sum, t1)
   168	#endif
   169		PTR_ADDU	src, src, 0x8
   170		andi	t8, src, 0x10
   171	
   172	.Loword_align:
   173		beqz	t8, .Lbegin_movement
   174		 LONG_SRL	t8, a1, 0x7
   175	
   176	#ifdef USE_DOUBLE
   177		ld	t0, 0x00(src)
   178		ld	t1, 0x08(src)
   179		ADDC(sum, t0)
   180		ADDC(sum, t1)
   181	#else
   182		CSUM_BIGCHUNK1(src, 0x00, sum, t0, t1, t3, t4)
   183	#endif
   184		LONG_SUBU	a1, a1, 0x10
   185		PTR_ADDU	src, src, 0x10
   186		LONG_SRL	t8, a1, 0x7
   187	
   188	.Lbegin_movement:
   189		beqz	t8, 1f
   190		 andi	t2, a1, 0x40
   191	
   192	.Lmove_128bytes:
   193	#ifdef CONFIG_CPU_HAS_PREFETCH
   194		/*
   195		 * Fetch the block two iterations ahead, but only while it lies
   196		 * inside the buffer: on non-coherent systems a line prefetched
   197		 * past the end could belong to a buffer the device still owns.
   198		 */
   199		sltiu	t5, t8, 3
   200		bnez	t5, 2f
   201		 nop
 > 202		pref	0, 0x100(src)
   203		pref	0, 0x120(src)
   204		pref	0, 0x140(src)
   205		pref	0, 0x160(src)
   206	2:
   207	#endif
   208		CSUM_BIGCHUNK(src, 0x00, sum, t0, t1, t3, t4)
   209		CSUM_BIGCHUNK(src, 0x20, sum, t0, t1, t3, t4)
   210		CSUM_BIGCHUNK(src, 0x40, sum, t0, t1, t3, t4)
   211		CSUM_BIGCHUNK(src, 0x60, sum, t0, t1, t3, t4)
   212		LONG_SUBU	t8, t8, 0x01
   213		.set	reorder				/* DADDI_WAR */
   214		PTR_ADDU	src, src, 0x80
   215		bnez	t8, .Lmove_128bytes
   216		.set	noreorder
   217	
   218	1:
   219		beqz	t2, 1f
   220		 andi	t2, a1, 0x20
   221	

--
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki