[PATCH] MIPS: lib: prefetch ahead in csum_partial()

[PATCH] MIPS: lib: prefetch ahead in csum_partial()

[Rosen Penev]

csum_partial() reads every byte of the buffer once. Cores like the 74K
have no hardware prefetcher, so every cache line is a demand miss, and
after a non-coherent DMA the received data is never in the cache. With
GRO enabled, validating the checksum of forwarded TCP traffic made
csum_partial() the top entry in the profile on a BCM4716 (74Kc at
480 MHz, no L2, ~80 cycles per DRAM miss): 13-16% of all cycles.

Prefetch the 128 byte block two iterations ahead in the main loop.
Unlike memcpy(), which disables prefetching on DMA_NONCOHERENT
entirely, only prefetch blocks that lie inside the buffer: the 74K does
not need a post-DMA invalidate, so a line prefetched past the end could
belong to a buffer the device is still writing, and the CPU would later
read the stale copy.

On an Asus RT-N16 (BCM4716, bgmac, single TCP stream unless noted,
iperf3 median of 3, A/B in the same boot by toggling the prefetch,
two rounds each):

                       without      with      (Mbit/s)
  local rx / tx        220 / 194    243 / 212
  routed up / down     169 / 140    187 / 154
  routed, 4 streams    65 / 79      69 / 84
  flow offload bidir   253          292

Assisted-by: LLM
Signed-off-by: Rosen Penev <rosenp@gmail.com>
---
 arch/mips/lib/csum_partial.S | 15 +++++++++++++++
 1 file changed, 15 insertions(+)

diff --git a/arch/mips/lib/csum_partial.S b/arch/mips/lib/csum_partial.S
index 3d2ff4118d79..f0ec47259ef4 100644
--- a/arch/mips/lib/csum_partial.S
+++ b/arch/mips/lib/csum_partial.S
@@ -190,6 +190,21 @@ EXPORT_SYMBOL(csum_partial)
 	 andi	t2, a1, 0x40
 
 .Lmove_128bytes:
+#ifdef CONFIG_CPU_HAS_PREFETCH
+	/*
+	 * Fetch the block two iterations ahead, but only while it lies
+	 * inside the buffer: on non-coherent systems a line prefetched
+	 * past the end could belong to a buffer the device still owns.
+	 */
+	sltiu	t5, t8, 3
+	bnez	t5, 2f
+	 nop
+	pref	0, 0x100(src)
+	pref	0, 0x120(src)
+	pref	0, 0x140(src)
+	pref	0, 0x160(src)
+2:
+#endif
 	CSUM_BIGCHUNK(src, 0x00, sum, t0, t1, t3, t4)
 	CSUM_BIGCHUNK(src, 0x20, sum, t0, t1, t3, t4)
 	CSUM_BIGCHUNK(src, 0x40, sum, t0, t1, t3, t4)
-- 
2.56.0

Re: [PATCH] MIPS: lib: prefetch ahead in csum_partial()

[kernel test robot]

Hi Rosen,

kernel test robot noticed the following build errors:

[auto build test ERROR on linus/master]
[also build test ERROR on v7.3-rc6 next-20261009]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/Rosen-Penev/MIPS-lib-prefetch-ahead-in-csum_partial/20261009-235442
base:   linus/master
patch link:    https://lore.kernel.org/r/20261010065442.23780-1-rosenp%40gmail.com
patch subject: [PATCH] MIPS: lib: prefetch ahead in csum_partial()
config: mips-rbtx49xx_defconfig (https://download.01.org/0day-ci/archive/20261013/202610131201.wWk11MLy-lkp@intel.com/config)
compiler: mips-linux-gcc (GCC) 16.1.0
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20261013/202610131201.wWk11MLy-lkp@intel.com/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <lkp@intel.com>
| Closes: https://lore.kernel.org/oe-kbuild-all/202610131201.wWk11MLy-lkp@intel.com/

All errors (new ones prefixed by >>):

   Assembler messages:
>> arch/mips/lib/csum_partial.S:202: Error: opcode not supported on this processor: r4600 (mips3) `pref 0,0x100($4)'
   arch/mips/lib/csum_partial.S:203: Error: opcode not supported on this processor: r4600 (mips3) `pref 0,0x120($4)'
   arch/mips/lib/csum_partial.S:204: Error: opcode not supported on this processor: r4600 (mips3) `pref 0,0x140($4)'
   arch/mips/lib/csum_partial.S:205: Error: opcode not supported on this processor: r4600 (mips3) `pref 0,0x160($4)'


vim +202 arch/mips/lib/csum_partial.S

   102	
   103		.text
   104		.set	noreorder
   105		.align	5
   106	LEAF(csum_partial)
   107	EXPORT_SYMBOL(csum_partial)
   108		move	sum, zero
   109		move	t7, zero
   110	
   111		sltiu	t8, a1, 0x8
   112		bnez	t8, .Lsmall_csumcpy		/* < 8 bytes to copy */
   113		 move	t2, a1
   114	
   115		andi	t7, src, 0x1			/* odd buffer? */
   116	
   117	.Lhword_align:
   118		beqz	t7, .Lword_align
   119		 andi	t8, src, 0x2
   120	
   121		lbu	t0, (src)
   122		LONG_SUBU	a1, a1, 0x1
   123	#ifdef __MIPSEL__
   124		sll	t0, t0, 8
   125	#endif
   126		ADDC(sum, t0)
   127		PTR_ADDU	src, src, 0x1
   128		andi	t8, src, 0x2
   129	
   130	.Lword_align:
   131		beqz	t8, .Ldword_align
   132		 sltiu	t8, a1, 56
   133	
   134		lhu	t0, (src)
   135		LONG_SUBU	a1, a1, 0x2
   136		ADDC(sum, t0)
   137		sltiu	t8, a1, 56
   138		PTR_ADDU	src, src, 0x2
   139	
   140	.Ldword_align:
   141		bnez	t8, .Ldo_end_words
   142		 move	t8, a1
   143	
   144		andi	t8, src, 0x4
   145		beqz	t8, .Lqword_align
   146		 andi	t8, src, 0x8
   147	
   148		LOAD32	t0, 0x00(src)
   149		LONG_SUBU	a1, a1, 0x4
   150		ADDC(sum, t0)
   151		PTR_ADDU	src, src, 0x4
   152		andi	t8, src, 0x8
   153	
   154	.Lqword_align:
   155		beqz	t8, .Loword_align
   156		 andi	t8, src, 0x10
   157	
   158	#ifdef USE_DOUBLE
   159		ld	t0, 0x00(src)
   160		LONG_SUBU	a1, a1, 0x8
   161		ADDC(sum, t0)
   162	#else
   163		lw	t0, 0x00(src)
   164		lw	t1, 0x04(src)
   165		LONG_SUBU	a1, a1, 0x8
   166		ADDC(sum, t0)
   167		ADDC(sum, t1)
   168	#endif
   169		PTR_ADDU	src, src, 0x8
   170		andi	t8, src, 0x10
   171	
   172	.Loword_align:
   173		beqz	t8, .Lbegin_movement
   174		 LONG_SRL	t8, a1, 0x7
   175	
   176	#ifdef USE_DOUBLE
   177		ld	t0, 0x00(src)
   178		ld	t1, 0x08(src)
   179		ADDC(sum, t0)
   180		ADDC(sum, t1)
   181	#else
   182		CSUM_BIGCHUNK1(src, 0x00, sum, t0, t1, t3, t4)
   183	#endif
   184		LONG_SUBU	a1, a1, 0x10
   185		PTR_ADDU	src, src, 0x10
   186		LONG_SRL	t8, a1, 0x7
   187	
   188	.Lbegin_movement:
   189		beqz	t8, 1f
   190		 andi	t2, a1, 0x40
   191	
   192	.Lmove_128bytes:
   193	#ifdef CONFIG_CPU_HAS_PREFETCH
   194		/*
   195		 * Fetch the block two iterations ahead, but only while it lies
   196		 * inside the buffer: on non-coherent systems a line prefetched
   197		 * past the end could belong to a buffer the device still owns.
   198		 */
   199		sltiu	t5, t8, 3
   200		bnez	t5, 2f
   201		 nop
 > 202		pref	0, 0x100(src)
   203		pref	0, 0x120(src)
   204		pref	0, 0x140(src)
   205		pref	0, 0x160(src)
   206	2:
   207	#endif
   208		CSUM_BIGCHUNK(src, 0x00, sum, t0, t1, t3, t4)
   209		CSUM_BIGCHUNK(src, 0x20, sum, t0, t1, t3, t4)
   210		CSUM_BIGCHUNK(src, 0x40, sum, t0, t1, t3, t4)
   211		CSUM_BIGCHUNK(src, 0x60, sum, t0, t1, t3, t4)
   212		LONG_SUBU	t8, t8, 0x01
   213		.set	reorder				/* DADDI_WAR */
   214		PTR_ADDU	src, src, 0x80
   215		bnez	t8, .Lmove_128bytes
   216		.set	noreorder
   217	
   218	1:
   219		beqz	t2, 1f
   220		 andi	t2, a1, 0x20
   221	

--
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki

Re: [PATCH] MIPS: lib: prefetch ahead in csum_partial()

[Maciej W. Rozycki]

On Fri, 9 Oct 2026, Rosen Penev wrote:

> diff --git a/arch/mips/lib/csum_partial.S b/arch/mips/lib/csum_partial.S
> index 3d2ff4118d79..f0ec47259ef4 100644
> --- a/arch/mips/lib/csum_partial.S
> +++ b/arch/mips/lib/csum_partial.S
> @@ -190,6 +190,21 @@ EXPORT_SYMBOL(csum_partial)
>  	 andi	t2, a1, 0x40
>  
>  .Lmove_128bytes:
> +#ifdef CONFIG_CPU_HAS_PREFETCH
> +	/*
> +	 * Fetch the block two iterations ahead, but only while it lies
> +	 * inside the buffer: on non-coherent systems a line prefetched
> +	 * past the end could belong to a buffer the device still owns.
> +	 */
> +	sltiu	t5, t8, 3
> +	bnez	t5, 2f
> +	 nop

 I don't like this manual delay slot scheduling, but it seems like a case 
for a larger clean-up of this file, so I'd be inclined to accept it.

> +	pref	0, 0x100(src)
> +	pref	0, 0x120(src)
> +	pref	0, 0x140(src)
> +	pref	0, 0x160(src)

 What about MIPSr6 though, which only has a span of -256..255 for the 
immediate offset in the machine instruction here?

  Maciej

Re: [PATCH] MIPS: lib: prefetch ahead in csum_partial()

[Maciej W. Rozycki]

On Tue, 13 Oct 2026, kernel test robot wrote:

> config: mips-rbtx49xx_defconfig (https://download.01.org/0day-ci/archive/20261013/202610131201.wWk11MLy-lkp@intel.com/config)
> compiler: mips-linux-gcc (GCC) 16.1.0
> reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20261013/202610131201.wWk11MLy-lkp@intel.com/reproduce)
> 
> If you fix the issue in a separate patch/commit (i.e. not just a new version of
> the same patch/commit), kindly add following tags
> | Reported-by: kernel test robot <lkp@intel.com>
> | Closes: https://lore.kernel.org/oe-kbuild-all/202610131201.wWk11MLy-lkp@intel.com/
> 
> All errors (new ones prefixed by >>):
> 
>    Assembler messages:
> >> arch/mips/lib/csum_partial.S:202: Error: opcode not supported on this processor: r4600 (mips3) `pref 0,0x100($4)'
>    arch/mips/lib/csum_partial.S:203: Error: opcode not supported on this processor: r4600 (mips3) `pref 0,0x120($4)'
>    arch/mips/lib/csum_partial.S:204: Error: opcode not supported on this processor: r4600 (mips3) `pref 0,0x140($4)'
>    arch/mips/lib/csum_partial.S:205: Error: opcode not supported on this processor: r4600 (mips3) `pref 0,0x160($4)'

 So oddly enough we're missing Tx49 support in the toolchain and we've 
only managed to survive with CPU_HAS_PREFETCH enabled for CPU_TX49XX 
because it also implies DMA_NONCOHERENT via MACH_TX49XX -> MACH_TXX9, 
which in turn disables stuff in arch/mips/lib/memcpy.S.

 Quite straighforward to fix in binutils since docs are available (as in: 
I have a copy), but do we have any users?  Geert?

  Maciej

Re: [PATCH] MIPS: lib: prefetch ahead in csum_partial()

[Geert Uytterhoeven]

Hi Maciej,

On Mon, 19 Oct 2026 at 01:26, Maciej W. Rozycki <macro@orcam.me.uk> wrote:
> On Tue, 13 Oct 2026, kernel test robot wrote:
> > config: mips-rbtx49xx_defconfig (https://download.01.org/0day-ci/archive/20261013/202610131201.wWk11MLy-lkp@intel.com/config)
> > compiler: mips-linux-gcc (GCC) 16.1.0
> > reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20261013/202610131201.wWk11MLy-lkp@intel.com/reproduce)
> >
> > If you fix the issue in a separate patch/commit (i.e. not just a new version of
> > the same patch/commit), kindly add following tags
> > | Reported-by: kernel test robot <lkp@intel.com>
> > | Closes: https://lore.kernel.org/oe-kbuild-all/202610131201.wWk11MLy-lkp@intel.com/
> >
> > All errors (new ones prefixed by >>):
> >
> >    Assembler messages:
> > >> arch/mips/lib/csum_partial.S:202: Error: opcode not supported on this processor: r4600 (mips3) `pref 0,0x100($4)'
> >    arch/mips/lib/csum_partial.S:203: Error: opcode not supported on this processor: r4600 (mips3) `pref 0,0x120($4)'
> >    arch/mips/lib/csum_partial.S:204: Error: opcode not supported on this processor: r4600 (mips3) `pref 0,0x140($4)'
> >    arch/mips/lib/csum_partial.S:205: Error: opcode not supported on this processor: r4600 (mips3) `pref 0,0x160($4)'
>
>  So oddly enough we're missing Tx49 support in the toolchain and we've
> only managed to survive with CPU_HAS_PREFETCH enabled for CPU_TX49XX
> because it also implies DMA_NONCOHERENT via MACH_TX49XX -> MACH_TXX9,
> which in turn disables stuff in arch/mips/lib/memcpy.S.
>
>  Quite straighforward to fix in binutils since docs are available (as in:
> I have a copy), but do we have any users?  Geert?

Yes, I still boot test rbtx4927 regularly.
I never touched binutils sources, though...

Gr{oetje,eeting}s,

                        Geert

-- 
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds

Re: [PATCH] MIPS: lib: prefetch ahead in csum_partial()

[Maciej W. Rozycki]

Hi Geert,

> >  So oddly enough we're missing Tx49 support in the toolchain and we've
> > only managed to survive with CPU_HAS_PREFETCH enabled for CPU_TX49XX
> > because it also implies DMA_NONCOHERENT via MACH_TX49XX -> MACH_TXX9,
> > which in turn disables stuff in arch/mips/lib/memcpy.S.
> >
> >  Quite straighforward to fix in binutils since docs are available (as in:
> > I have a copy), but do we have any users?  Geert?
> 
> Yes, I still boot test rbtx4927 regularly.

 Thanks for confirming.

> I never touched binutils sources, though...

 No worries, this seems straightforward: just PREF, MADD/U plus [D]MULT/U 
extensions, and DERET/SDBBP debug stuff, much of which shared with the 
Tx39 we already support, so just a matter of sticking another flag onto 
the existing opcode table entries.

 I'd estimate it as one day worth of engineering effort plus maybe another 
one for machines to work through verification, for the binutils side and 
the associated changes needed for GCC, i.e. a `-march=' option update plus 
maybe enabling a couple of existing machine description patterns for the 
Tx49 too.  Would you be able to run GCC verification with your system?

 So I guess I could squeeze this in after the oustanding Allegrex support 
review, hopefully to start this week.

  Maciej

Re: [PATCH] MIPS: lib: prefetch ahead in csum_partial()

[Geert Uytterhoeven]

Hi Maciej,

On Mon, 19 Oct 2026 at 16:47, Maciej W. Rozycki <macro@orcam.me.uk> wrote:
> > >  So oddly enough we're missing Tx49 support in the toolchain and we've
> > > only managed to survive with CPU_HAS_PREFETCH enabled for CPU_TX49XX
> > > because it also implies DMA_NONCOHERENT via MACH_TX49XX -> MACH_TXX9,
> > > which in turn disables stuff in arch/mips/lib/memcpy.S.
> > >
> > >  Quite straighforward to fix in binutils since docs are available (as in:
> > > I have a copy), but do we have any users?  Geert?
> >
> > Yes, I still boot test rbtx4927 regularly.
>
>  Thanks for confirming.
>
> > I never touched binutils sources, though...
>
>  No worries, this seems straightforward: just PREF, MADD/U plus [D]MULT/U
> extensions, and DERET/SDBBP debug stuff, much of which shared with the
> Tx39 we already support, so just a matter of sticking another flag onto
> the existing opcode table entries.
>
>  I'd estimate it as one day worth of engineering effort plus maybe another
> one for machines to work through verification, for the binutils side and
> the associated changes needed for GCC, i.e. a `-march=' option update plus
> maybe enabling a couple of existing machine description patterns for the
> Tx49 too.  Would you be able to run GCC verification with your system?

The rbtx4927 is about the slowest system I still do regular testing on.
It uses a Debian 8 nfsroot (over 10 Mbps Ethernet), which is the latest
Debian release still supporting the instruction subset suppported
by TX49XX.
So as long as no native compilation of large packages is required,
it can be done.  I can also give you remote access?

Thanks!

Gr{oetje,eeting}s,

                        Geert

-- 
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds

Re: [PATCH] MIPS: lib: prefetch ahead in csum_partial()

[Maciej W. Rozycki]

Hi Geert,

> >  I'd estimate it as one day worth of engineering effort plus maybe another
> > one for machines to work through verification, for the binutils side and
> > the associated changes needed for GCC, i.e. a `-march=' option update plus
> > maybe enabling a couple of existing machine description patterns for the
> > Tx49 too.  Would you be able to run GCC verification with your system?
> 
> The rbtx4927 is about the slowest system I still do regular testing on.

 Speed is of no essence, I've recently run regression-testing with a R3k 
DECstation, a VAXstation (with NetBSD), and previously, as a day job, with 
a crippled ARM board well below the capabilities of the former systems.  
Depending on the components chosen and the amount of breakage triggered by 
the tests themselves (timeouts!) it could take from several hours to a few 
days.

> It uses a Debian 8 nfsroot (over 10 Mbps Ethernet), which is the latest
> Debian release still supporting the instruction subset suppported
> by TX49XX.

 I run most of my target systems NFS-rooted as well (over 100Mbps FDDI for 
a change).  Little is required from the target environment: an SSH server 
and a POSIX shell, and then either an NFS client for direct build system 
access or rsync for remote sysroot maintenance (in which case `rm' command 
is needed too to discard test cases copied over, once executed).

> So as long as no native compilation of large packages is required,
> it can be done.  I can also give you remote access?

 There is no native compilation at all, only remote execution over SSH, 
driven from the build system by DejaGNU.

 As for remote access, that would be an option, perhaps an easy one since 
I've done it before with someone else, as long as you can offer temporary 
storage for the sysroot to operate the target system with (~225MiB for 
o32, ~285MiB for n64, indicative) and some network bandwidth.

  Maciej