From: Anton Blanchard <anton@samba.org>
To: benh@kernel.crashing.org, paulus@samba.org,
michael@ellerman.id.au, mikey@neuling.org
Cc: linuxppc-dev@lists.ozlabs.org
Subject: [PATCH] powerpc: Optimise the 64bit optimised __clear_user
Date: Mon, 4 Jun 2012 17:58:58 +1000 [thread overview]
Message-ID: <20120604175858.38dac554@kryten> (raw)
I blame Mikey for this. He elevated my slightly dubious testcase:
# dd if=/dev/zero of=/dev/null bs=1M count=10000
to benchmark status. And naturally we need to be number 1 at creating
zeros. So lets improve __clear_user some more.
As Paul suggests we can use dcbz for large lengths. This patch gets
the destination 128 byte aligned then uses dcbz on whole cachelines.
Before:
10485760000 bytes (10 GB) copied, 0.414744 s, 25.3 GB/s
After:
10485760000 bytes (10 GB) copied, 0.268597 s, 39.0 GB/s
39 GB/s, a new record.
Signed-off-by: Anton Blanchard <anton@samba.org>
---
Index: linux-build/arch/powerpc/lib/string_64.S
===================================================================
--- linux-build.orig/arch/powerpc/lib/string_64.S 2012-06-04 16:18:56.351604302 +1000
+++ linux-build/arch/powerpc/lib/string_64.S 2012-06-04 16:47:10.538500871 +1000
@@ -78,7 +78,7 @@ _GLOBAL(__clear_user)
blt .Lshort_clear
mr r8,r3
mtocrf 0x01,r6
- clrldi r6,r6,(64-3)
+ clrldi r7,r6,(64-3)
/* Get the destination 8 byte aligned */
bf cr7*4+3,1f
@@ -93,11 +93,16 @@ err1; sth r0,0(r3)
err1; stw r0,0(r3)
addi r3,r3,4
-3: sub r4,r4,r6
- srdi r6,r4,5
+3: sub r4,r4,r7
+
cmpdi r4,32
+ cmpdi cr1,r4,512
blt .Lshort_clear
- mtctr r6
+ bgt cr1,.Llong_clear
+
+.Lmedium_clear:
+ srdi r7,r4,5
+ mtctr r7
/* Do 32 byte chunks */
4:
@@ -139,3 +144,52 @@ err1; stb r0,0(r3)
10: li r3,0
blr
+
+.Llong_clear:
+ /* Destination is 8 byte aligned, need to get it 128 byte aligned */
+ bf cr7*4+0,11f
+err2; std r0,0(r3)
+ addi r3,r3,8
+ addi r4,r4,-8
+
+11: srdi r6,r6,4
+ mtocrf 0x01,r6
+
+ bf cr7*4+3,12f
+err2; std r0,0(r3)
+err2; std r0,8(r3)
+ addi r3,r3,16
+ addi r4,r4,-16
+
+12: bf cr7*4+2,13f
+err2; std r0,0(r3)
+err2; std r0,8(r3)
+err2; std r0,16(r3)
+err2; std r0,24(r3)
+ addi r3,r3,32
+ addi r4,r4,-32
+
+13: bf cr7*4+1,14f
+err2; std r0,0(r3)
+err2; std r0,8(r3)
+err2; std r0,16(r3)
+err2; std r0,24(r3)
+err2; std r0,32(r3)
+err2; std r0,40(r3)
+err2; std r0,48(r3)
+err2; std r0,56(r3)
+ addi r3,r3,64
+ addi r4,r4,-64
+
+14: srdi r6,r4,7
+ mtctr r6
+
+15:
+err2; dcbz r0,r3
+ addi r3,r3,128
+ addi r4,r4,-128
+ bdnz 15b
+
+ cmpdi r4,32
+ blt .Lshort_clear
+ b .Lmedium_clear
next reply other threads:[~2012-06-04 7:58 UTC|newest]
Thread overview: 63+ messages / expand[flat|nested] mbox.gz Atom feed top
2012-06-04 7:58 Anton Blanchard [this message]
2012-06-04 13:12 ` [PATCH] powerpc: Optimise the 64bit optimised __clear_user Olof Johansson
2012-06-04 14:44 ` Kumar Gala
2012-06-05 2:02 ` Anton Blanchard
2012-06-06 16:40 ` Segher Boessenkool
2012-06-06 21:20 ` Benjamin Herrenschmidt
2012-06-07 6:05 ` Michael Neuling
2012-06-07 6:07 ` Michael Ellerman
2012-06-07 6:12 ` Michael Neuling
2012-06-07 6:18 ` Michael Ellerman
2012-06-07 23:52 ` [PATCH] powerpc: Fix sldi to take literal not register name Michael Neuling
2012-06-08 0:19 ` Benjamin Herrenschmidt
2012-06-07 6:39 ` [PATCH] powerpc: Optimise the 64bit optimised __clear_user Benjamin Herrenschmidt
2012-06-07 8:41 ` David Laight
2012-06-08 11:36 ` [PATCH 0/15] powerpc: convert GPR usage to %r0-31 and R0-31 Michael Neuling
2012-06-08 11:36 ` [PATCH 7/15] powerpc: merge STK_REG/PARAM/FRAMESIZE Michael Neuling
2012-06-08 11:36 ` [PATCH 10/15] powerpc: fixes for instructions not using correct register naming Michael Neuling
2012-06-08 11:36 ` [PATCH 8/15] powerpc: change mtcrf to use real register names Michael Neuling
2012-06-08 11:36 ` [PATCH 15/15] powerpc: enforce usage of R0-R31 where possible Michael Neuling
2012-06-08 11:36 ` [PATCH 1/15] powerpc: Add defines for R0-R31 Michael Neuling
2012-06-08 11:36 ` [PATCH 5/15] powerpc: convert to %r for all GPR usage Michael Neuling
2012-06-08 21:54 ` Jesse Larrew
2012-06-14 3:25 ` Michael Neuling
2012-06-14 6:50 ` Jesse Larrew
2012-06-08 11:36 ` [PATCH 4/15] powerpc: Fix sldi to take literal not register name Michael Neuling
2012-06-08 11:36 ` [PATCH 14/15] powerpc: Introduce new __REG_R macros Michael Neuling
2012-06-08 11:36 ` [PATCH 2/15] powerpc: modify macro ready for %r0 register change Michael Neuling
2012-06-08 11:36 ` [PATCH 13/15] powerpc: start using ___PPC_RA/B/S/T where necessary Michael Neuling
2012-06-08 11:36 ` [PATCH 12/15] powerpc: introduce new ___PPC_RA/B/S/T macros Michael Neuling
2012-06-08 11:36 ` [PATCH 9/15] powerpc: change LOAD_REG_ADDR to use real register names Michael Neuling
2012-06-08 11:36 ` [PATCH 6/15] powerpc/pasemi: move lbz/stbciz to ppc-opcode.h Michael Neuling
2012-06-08 11:36 ` [PATCH 3/15] powerpc: fix usage of register macros getting ready for %r0 change Michael Neuling
2012-06-08 11:36 ` [PATCH 11/15] powerpc: fix VSX macros so register names aren't wrapped Michael Neuling
[not found] ` <20120608113605.80B73D43B2B__2857.47079547054$1339155968$gmane$org@localhost.localdomain>
2012-06-08 12:12 ` [PATCH 7/15] powerpc: merge STK_REG/PARAM/FRAMESIZE Andreas Schwab
[not found] ` <20120608113605.91B88D43B2F__24549.446340143$1339156128$gmane$org@localhost.localdomain>
2012-06-08 12:15 ` [PATCH 10/15] powerpc: fixes for instructions not using correct register naming Andreas Schwab
2012-06-08 22:42 ` Benjamin Herrenschmidt
2012-06-09 6:53 ` Andreas Schwab
2012-06-09 7:17 ` Benjamin Herrenschmidt
2012-06-09 9:39 ` Andreas Schwab
2012-06-14 6:15 ` [PATCH 0/18] powerpc: convert GPR usage to %r0-31 and R0-31 Michael Neuling
2012-06-14 6:15 ` [PATCH 1/18] powerpc: Add defines for R0-R31 Michael Neuling
2012-06-14 6:15 ` [PATCH 2/18] powerpc: modify macro ready for %r0 register change Michael Neuling
2012-06-14 6:15 ` [PATCH 3/18] powerpc: fix usage of register macros getting ready for %r0 change Michael Neuling
2012-06-14 6:15 ` [PATCH 4/18] powerpc: Fix sldi to take literal not register name Michael Neuling
2012-06-14 6:15 ` [PATCH 5/18] powerpc: convert to %r for all GPR usage Michael Neuling
2012-06-14 6:15 ` [PATCH 6/18] powerpc/pasemi: move lbz/stbciz to ppc-opcode.h Michael Neuling
2012-06-14 6:15 ` [PATCH 7/18] powerpc: merge STK_REG/PARAM/FRAMESIZE Michael Neuling
2012-06-14 6:15 ` [PATCH 8/18] powerpc: merge VCPU_GPR Michael Neuling
2012-06-14 6:15 ` [PATCH 9/18] powerpc: change mtcrf to use real register names Michael Neuling
2012-06-14 6:15 ` [PATCH 10/18] powerpc: change LOAD_REG_ADDR " Michael Neuling
2012-06-14 6:15 ` [PATCH 11/18] powerpc: fixes for instructions not using correct register naming Michael Neuling
2012-06-14 6:15 ` [PATCH 12/18] powerpc: fix VSX macros so register names aren't wrapped Michael Neuling
2012-06-14 6:15 ` [PATCH 13/18] powerpc: introduce new ___PPC_RA/B/S/T macros Michael Neuling
2012-06-14 6:15 ` [PATCH 14/18] powerpc: start using ___PPC_RA/B/S/T where necessary Michael Neuling
2012-06-14 6:15 ` [PATCH 15/18] powerpc: Introduce new __REG_R macros Michael Neuling
2012-06-14 6:15 ` [PATCH 16/18] powerpc: enforce usage of R0-R31 where possible Michael Neuling
2012-06-14 6:15 ` [PATCH 17/18] powerpc: Add defines for RA 0-R31 Michael Neuling
2012-06-14 6:15 ` [PATCH 18/18] powerpc: enforce usage of RA 0-R31 where possible Michael Neuling
2012-06-07 3:04 ` [PATCH] powerpc: Optimise the 64bit optimised __clear_user Paul Mackerras
2012-06-07 17:51 ` Scott Wood
2012-06-08 0:51 ` Benjamin Herrenschmidt
2012-06-08 7:34 ` Andreas Schwab
2012-06-07 0:30 ` Olof Johansson
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20120604175858.38dac554@kryten \
--to=anton@samba.org \
--cc=benh@kernel.crashing.org \
--cc=linuxppc-dev@lists.ozlabs.org \
--cc=michael@ellerman.id.au \
--cc=mikey@neuling.org \
--cc=paulus@samba.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.