From: Christoph Lameter <cl@linux-foundation.org>
To: Tejun Heo <tj@kernel.org>
Cc: linux-kernel@vger.kernel.org
Cc: Mel Gorman <mel@csn.ul.ie>
Cc: Pekka Enberg <penberg@cs.helsinki.fi>
Cc: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>
Subject: [this_cpu_xx V8 14/16] x86 percpu xchg operation
Date: Fri, 18 Dec 2009 16:26:31 -0600 [thread overview]
Message-ID: <20091218222655.192407545@quilx.com> (raw)
In-Reply-To: 20091218222617.384355422@quilx.com
[-- Attachment #1: percpu_add_xchg_x86 --]
[-- Type: text/plain, Size: 4101 bytes --]
Xchg on x86 implies LOCK semantics and therefore is slow. Approximate it
using this_cpu_cmpxchg().
Signed-off-by: Christoph Lameter <cl@linux-foundation.org>
---
arch/x86/include/asm/percpu.h | 21 +++++++++++++++++++++
1 file changed, 21 insertions(+)
Index: linux-2.6/arch/x86/include/asm/percpu.h
===================================================================
--- linux-2.6.orig/arch/x86/include/asm/percpu.h 2009-12-18 15:49:09.000000000 -0600
+++ linux-2.6/arch/x86/include/asm/percpu.h 2009-12-18 15:50:12.000000000 -0600
@@ -156,6 +156,15 @@ do { \
} \
})
+#define this_cpu_xchg_x86(var, new) \
+({ __typeof(*var) __tmp_var,__tmp_old; \
+ do { \
+ __tmp_old = __this_cpu_read(var); \
+ __tmp_var = __this_cpu_cmpxchg(var, __tmp_old, new); \
+ } while (__tmp_var != __tmp_old); \
+ __tmp_old; \
+})
+
/*
* percpu_read() makes gcc load the percpu variable every time it is
* accessed while percpu_read_stable() allows the value to be cached.
@@ -204,6 +213,9 @@ do { \
#define __this_cpu_cmpxchg_1(pcp, old,new) cmpxchg_local(__this_cpu_ptr(pcp), old, new)
#define __this_cpu_cmpxchg_2(pcp, old,new) cmpxchg_local(__this_cpu_ptr(pcp), old, new)
#define __this_cpu_cmpxchg_4(pcp, old,new) cmpxchg_local(__this_cpu_ptr(pcp), old, new)
+#define __this_cpu_xchg_1(pcp, new) this_cpu_xchg_x86((pcp), new)
+#define __this_cpu_xchg_2(pcp, new) this_cpu_xchg_x86((pcp), new)
+#define __this_cpu_xchg_4(pcp, new) this_cpu_xchg_x86((pcp), new)
#define this_cpu_read_1(pcp) percpu_from_op("mov", (pcp), "m"(pcp))
#define this_cpu_read_2(pcp) percpu_from_op("mov", (pcp), "m"(pcp))
@@ -232,6 +244,9 @@ do { \
#define this_cpu_cmpxchg_1(pcp, old,new) cmpxchg_local(this_cpu_ptr(pcp), old, new)
#define this_cpu_cmpxchg_2(pcp, old,new) cmpxchg_local(this_cpu_ptr(pcp), old, new)
#define this_cpu_cmpxchg_4(pcp, old,new) cmpxchg_local(this_cpu_ptr(pcp), old, new)
+#define this_cpu_xchg_1(pcp, new) this_cpu_xchg_x86((pcp), new)
+#define this_cpu_xchg_2(pcp, new) this_cpu_xchg_x86((pcp), new)
+#define this_cpu_xchg_4(pcp, new) this_cpu_xchg_x86((pcp), new)
#define irqsafe_cpu_add_1(pcp, val) percpu_to_op("add", (pcp), val)
#define irqsafe_cpu_add_2(pcp, val) percpu_to_op("add", (pcp), val)
@@ -254,6 +269,9 @@ do { \
#define irqsafe_cpu_cmpxchg_1(pcp, old,new) cmpxchg_local(this_cpu_ptr(pcp), old, new)
#define irqsafe_cpu_cmpxchg_2(pcp, old,new) cmpxchg_local(this_cpu_ptr(pcp), old, new)
#define irqsafe_cpu_cmpxchg_4(pcp, old,new) cmpxchg_local(this_cpu_ptr(pcp), old, new)
+#define irqsafe_cpu_xchg_1(pcp, new) this_cpu_xchg_x86((pcp), new)
+#define irqsafe_cpu_xchg_2(pcp, new) this_cpu_xchg_x86((pcp), new)
+#define irqsafe_cpu_xchg_4(pcp, new) this_cpu_xchg_x86((pcp), new)
/*
* Per cpu atomic 64 bit operations are only available under 64 bit.
@@ -269,6 +287,7 @@ do { \
#define __this_cpu_or_8(pcp, val) percpu_to_op("or", (pcp), val)
#define __this_cpu_xor_8(pcp, val) percpu_to_op("xor", (pcp), val)
#define __this_cpu_cmpxchg_8(pcp, old,new) cmpxchg_local(__this_cpu_ptr(pcp), old, new)
+#define __this_cpu_xchg_8(pcp, new) this_cpu_xchg_x86((pcp), new)
#define this_cpu_read_8(pcp) percpu_from_op("mov", (pcp), "m"(pcp))
#define this_cpu_write_8(pcp, val) percpu_to_op("mov", (pcp), val)
@@ -279,6 +298,7 @@ do { \
#define this_cpu_or_8(pcp, val) percpu_to_op("or", (pcp), val)
#define this_cpu_xor_8(pcp, val) percpu_to_op("xor", (pcp), val)
#define this_cpu_cmpxchg_8(pcp, old,new) cmpxchg_local(this_cpu_ptr(pcp), old, new)
+#define this_cpu_xchg_8(pcp, new) this_cpu_xchg_x86((pcp), new)
#define irqsafe_cpu_add_8(pcp, val) percpu_to_op("add", (pcp), val)
#define irqsafe_cpu_inc_8(pcp) percpu_var_op("inc", (pcp))
@@ -287,6 +307,7 @@ do { \
#define irqsafe_cpu_or_8(pcp, val) percpu_to_op("or", (pcp), val)
#define irqsafe_cpu_xor_8(pcp, val) percpu_to_op("xor", (pcp), val)
#define irqsafe_cpu_cmpxchg_8(pcp, old,new) cmpxchg_local(this_cpu_ptr(pcp), old, new)
+#define irqsafe_cpu_xchg_8(pcp, new) this_cpu_xchg_x86((pcp), new)
#endif
--
next prev parent reply other threads:[~2009-12-19 0:28 UTC|newest]
Thread overview: 38+ messages / expand[flat|nested] mbox.gz Atom feed top
2009-12-18 22:26 [this_cpu_xx V8 00/16] Per cpu atomics in core allocators, cleanup and more this_cpu_ops Christoph Lameter
2009-12-18 22:26 ` [this_cpu_xx V8 01/16] this_cpu_ops: page allocator conversion Christoph Lameter
2009-12-18 22:26 ` [this_cpu_xx V8 02/16] this_cpu ops: Remove pageset_notifier Christoph Lameter
2009-12-18 22:26 ` [this_cpu_xx V8 03/16] Use this_cpu operations in slub Christoph Lameter
2009-12-20 9:11 ` Pekka Enberg
2009-12-18 22:26 ` [this_cpu_xx V8 04/16] SLUB: Get rid of dynamic DMA kmalloc cache allocation Christoph Lameter
2009-12-18 22:26 ` [this_cpu_xx V8 05/16] this_cpu: Remove slub kmem_cache fields Christoph Lameter
2009-12-18 22:26 ` [this_cpu_xx V8 06/16] Make slub statistics use this_cpu_inc Christoph Lameter
2009-12-18 22:26 ` [this_cpu_xx V8 07/16] Module handling: Use this_cpu_xx to dynamically allocate counters Christoph Lameter
2009-12-21 7:47 ` Tejun Heo
2009-12-21 7:59 ` Tejun Heo
2009-12-21 8:19 ` Tejun Heo
2009-12-21 23:28 ` Rusty Russell
2009-12-22 0:02 ` Tejun Heo
2009-12-22 16:17 ` Christoph Lameter
2009-12-22 15:58 ` Christoph Lameter
2009-12-22 15:57 ` Christoph Lameter
2009-12-23 2:07 ` Tejun Heo
2010-01-04 17:22 ` Christoph Lameter
2010-01-04 17:51 ` Mathieu Desnoyers
2009-12-18 22:26 ` [this_cpu_xx V8 08/16] Remove cpu_local_xx macros Christoph Lameter
2009-12-18 22:26 ` [this_cpu_xx V8 09/16] Allow arch to provide inc/dec functionality for each size separately Christoph Lameter
2009-12-21 7:25 ` Tejun Heo
2009-12-22 15:56 ` Christoph Lameter
2009-12-23 2:08 ` Tejun Heo
2009-12-18 22:26 ` [this_cpu_xx V8 10/16] Support generating inc/dec for this_cpu_inc/dec Christoph Lameter
2009-12-18 22:26 ` [this_cpu_xx V8 11/16] Generic support for this_cpu_cmpxchg Christoph Lameter
2009-12-19 14:45 ` Mathieu Desnoyers
2009-12-22 15:54 ` Christoph Lameter
2009-12-22 17:24 ` Mathieu Desnoyers
2010-01-04 17:21 ` Christoph Lameter
2010-01-05 22:29 ` Mathieu Desnoyers
2010-01-05 22:35 ` Christoph Lameter
2009-12-18 22:26 ` [this_cpu_xx V8 12/16] Add percpu cmpxchg operations Christoph Lameter
2009-12-18 22:26 ` [this_cpu_xx V8 13/16] Generic support for this_cpu_xchg Christoph Lameter
2009-12-18 22:26 ` Christoph Lameter [this message]
2009-12-18 22:26 ` [this_cpu_xx V8 15/16] Generic support for this_cpu_add_return() Christoph Lameter
2009-12-18 22:26 ` [this_cpu_xx V8 16/16] x86 support for this_cpu_add_return Christoph Lameter
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20091218222655.192407545@quilx.com \
--to=cl@linux-foundation.org \
--cc=linux-kernel@vger.kernel.org \
--cc=tj@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox