From: guoren@kernel.org
To: guoren@kernel.org, arnd@arndb.de, palmer@dabbelt.com,
mark.rutland@arm.com, will@kernel.org, peterz@infradead.org,
boqun.feng@gmail.com
Cc: linux-arch@vger.kernel.org, linux-kernel@vger.kernel.org,
linux-riscv@lists.infradead.org,
Guo Ren <guoren@linux.alibaba.com>
Subject: [PATCH V2 3/3] riscv: atomic: Optimize memory barrier semantics of LRSC-pairs
Date: Tue, 12 Apr 2022 11:49:57 +0800 [thread overview]
Message-ID: <20220412034957.1481088-4-guoren@kernel.org> (raw)
In-Reply-To: <20220412034957.1481088-1-guoren@kernel.org>
From: Guo Ren <guoren@linux.alibaba.com>
The current implementation is the same with 8e86f0b409a4 ("arm64:
atomics: fix use of acquire + release for full barrier semantics").
RISC-V could combine acquire and release into the AMO instructions
and it could reduce the cost of instruction in performance. Here
are the reasons for optimization:
- Reduce one extra fence instruction
- The "LR/SC" instruction with "acquire and release" operation is
less cost than ACQUIRE_BARRIER/RELEASE_BARRIER which used
precedes-loads/subsequent-stores prohibit to protect only LR/SC
self-instruction.
- Putting acquire/release barrier into the loop shouldn't cost
extra performance problems from the micro-arch design view.
Because LR and SC are sequential in the loop by RVWMO rules.
Signed-off-by: Guo Ren <guoren@linux.alibaba.com>
Signed-off-by: Guo Ren <guoren@kernel.org>
Cc: Palmer Dabbelt <palmer@dabbelt.com>
Cc: Mark Rutland <mark.rutland@arm.com>
---
arch/riscv/include/asm/atomic.h | 6 ++----
arch/riscv/include/asm/cmpxchg.h | 18 ++++++------------
2 files changed, 8 insertions(+), 16 deletions(-)
diff --git a/arch/riscv/include/asm/atomic.h b/arch/riscv/include/asm/atomic.h
index 20ce8b83bc18..4aaf5b01e7c6 100644
--- a/arch/riscv/include/asm/atomic.h
+++ b/arch/riscv/include/asm/atomic.h
@@ -382,9 +382,8 @@ static __always_inline int arch_atomic_sub_if_positive(atomic_t *v, int offset)
"0: lr.w %[p], %[c]\n"
" sub %[rc], %[p], %[o]\n"
" bltz %[rc], 1f\n"
- " sc.w.rl %[rc], %[rc], %[c]\n"
+ " sc.w.aqrl %[rc], %[rc], %[c]\n"
" bnez %[rc], 0b\n"
- " fence rw, rw\n"
"1:\n"
: [p]"=&r" (prev), [rc]"=&r" (rc), [c]"+A" (v->counter)
: [o]"r" (offset)
@@ -404,9 +403,8 @@ static __always_inline s64 arch_atomic64_sub_if_positive(atomic64_t *v, s64 offs
"0: lr.d %[p], %[c]\n"
" sub %[rc], %[p], %[o]\n"
" bltz %[rc], 1f\n"
- " sc.d.rl %[rc], %[rc], %[c]\n"
+ " sc.d.aqrl %[rc], %[rc], %[c]\n"
" bnez %[rc], 0b\n"
- " fence rw, rw\n"
"1:\n"
: [p]"=&r" (prev), [rc]"=&r" (rc), [c]"+A" (v->counter)
: [o]"r" (offset)
diff --git a/arch/riscv/include/asm/cmpxchg.h b/arch/riscv/include/asm/cmpxchg.h
index 1af8db92250b..dfb51c98324d 100644
--- a/arch/riscv/include/asm/cmpxchg.h
+++ b/arch/riscv/include/asm/cmpxchg.h
@@ -215,9 +215,8 @@
__asm__ __volatile__ ( \
"0: lr.w %0, %2\n" \
" bne %0, %z3, 1f\n" \
- " sc.w %1, %z4, %2\n" \
+ " sc.w.aq %1, %z4, %2\n" \
" bnez %1, 0b\n" \
- RISCV_ACQUIRE_BARRIER \
"1:\n" \
: "=&r" (__ret), "=&r" (__rc), "+A" (*__ptr) \
: "rJ" ((long)__old), "rJ" (__new) \
@@ -227,9 +226,8 @@
__asm__ __volatile__ ( \
"0: lr.d %0, %2\n" \
" bne %0, %z3, 1f\n" \
- " sc.d %1, %z4, %2\n" \
+ " sc.d.aq %1, %z4, %2\n" \
" bnez %1, 0b\n" \
- RISCV_ACQUIRE_BARRIER \
"1:\n" \
: "=&r" (__ret), "=&r" (__rc), "+A" (*__ptr) \
: "rJ" (__old), "rJ" (__new) \
@@ -259,8 +257,7 @@
switch (size) { \
case 4: \
__asm__ __volatile__ ( \
- RISCV_RELEASE_BARRIER \
- "0: lr.w %0, %2\n" \
+ "0: lr.w.rl %0, %2\n" \
" bne %0, %z3, 1f\n" \
" sc.w %1, %z4, %2\n" \
" bnez %1, 0b\n" \
@@ -271,8 +268,7 @@
break; \
case 8: \
__asm__ __volatile__ ( \
- RISCV_RELEASE_BARRIER \
- "0: lr.d %0, %2\n" \
+ "0: lr.d.rl %0, %2\n" \
" bne %0, %z3, 1f\n" \
" sc.d %1, %z4, %2\n" \
" bnez %1, 0b\n" \
@@ -307,9 +303,8 @@
__asm__ __volatile__ ( \
"0: lr.w %0, %2\n" \
" bne %0, %z3, 1f\n" \
- " sc.w.rl %1, %z4, %2\n" \
+ " sc.w.aqrl %1, %z4, %2\n" \
" bnez %1, 0b\n" \
- " fence rw, rw\n" \
"1:\n" \
: "=&r" (__ret), "=&r" (__rc), "+A" (*__ptr) \
: "rJ" ((long)__old), "rJ" (__new) \
@@ -319,9 +314,8 @@
__asm__ __volatile__ ( \
"0: lr.d %0, %2\n" \
" bne %0, %z3, 1f\n" \
- " sc.d.rl %1, %z4, %2\n" \
+ " sc.d.aqrl %1, %z4, %2\n" \
" bnez %1, 0b\n" \
- " fence rw, rw\n" \
"1:\n" \
: "=&r" (__ret), "=&r" (__rc), "+A" (*__ptr) \
: "rJ" (__old), "rJ" (__new) \
--
2.25.1
next prev parent reply other threads:[~2022-04-12 3:50 UTC|newest]
Thread overview: 21+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-04-12 3:49 [PATCH V2 0/3] riscv: atomic: Optimize AMO instructions usage guoren
2022-04-12 3:49 ` [PATCH V2 1/3] riscv: atomic: Cleanup unnecessary definition guoren
2022-04-12 3:49 ` [PATCH V2 2/3] riscv: atomic: Optimize acquire and release for AMO operations guoren
2022-04-12 3:49 ` guoren [this message]
2022-04-13 15:46 ` [PATCH V2 0/3] riscv: atomic: Optimize AMO instructions usage Boqun Feng
2022-04-16 16:49 ` Guo Ren
2022-04-17 2:26 ` Boqun Feng
2022-04-17 4:51 ` Guo Ren
2022-04-17 6:30 ` Boqun Feng
2022-04-17 6:45 ` Guo Ren
2022-04-19 17:12 ` Dan Lustig
2022-04-20 5:33 ` Guo Ren
2022-04-20 17:03 ` Dan Lustig
2022-04-21 9:39 ` Guo Ren
2022-04-21 22:56 ` Boqun Feng
2022-04-22 1:56 ` Guo Ren
2022-04-22 3:11 ` Boqun Feng
2022-04-24 7:52 ` Guo Ren
2022-04-18 23:41 ` Andrea Parri
2022-04-19 17:13 ` Dan Lustig
2022-04-24 8:33 ` Guo Ren
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20220412034957.1481088-4-guoren@kernel.org \
--to=guoren@kernel.org \
--cc=arnd@arndb.de \
--cc=boqun.feng@gmail.com \
--cc=guoren@linux.alibaba.com \
--cc=linux-arch@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-riscv@lists.infradead.org \
--cc=mark.rutland@arm.com \
--cc=palmer@dabbelt.com \
--cc=peterz@infradead.org \
--cc=will@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox