* [PATCH 1/2] powerpc/powernv: Add PE to its own PELTV
From: Gavin Shan @ 2013-11-04 8:32 UTC (permalink / raw)
To: linuxppc-dev; +Cc: Gavin Shan, stable
We need add PE to its own PELTV. Otherwise, the errors originated
from the PE might contribute to other PEs. In the result, we can't
clear up the error successfully even we're checking and clearing
errors during access to PCI config space.
Cc: stable@vger.kernel.org
Reported-by: kalshett@in.ibm.com
Signed-off-by: Gavin Shan <shangw@linux.vnet.ibm.com>
---
arch/powerpc/platforms/powernv/pci-ioda.c | 12 +++++++++++-
1 file changed, 11 insertions(+), 1 deletion(-)
diff --git a/arch/powerpc/platforms/powernv/pci-ioda.c b/arch/powerpc/platforms/powernv/pci-ioda.c
index c639af7..198566e 100644
--- a/arch/powerpc/platforms/powernv/pci-ioda.c
+++ b/arch/powerpc/platforms/powernv/pci-ioda.c
@@ -163,13 +163,23 @@ static int pnv_ioda_configure_pe(struct pnv_phb *phb, struct pnv_ioda_pe *pe)
rid_end = pe->rid + 1;
}
- /* Associate PE in PELT */
+ /*
+ * Associate PE in PELT. We need add the PE into the
+ * corresponding PELT-V as well. Otherwise, the error
+ * originated from the PE might contribute to other
+ * PEs.
+ */
rc = opal_pci_set_pe(phb->opal_id, pe->pe_number, pe->rid,
bcomp, dcomp, fcomp, OPAL_MAP_PE);
if (rc) {
pe_err(pe, "OPAL error %ld trying to setup PELT table\n", rc);
return -ENXIO;
}
+
+ rc = opal_pci_set_peltv(phb->opal_id, pe->pe_number,
+ pe->pe_number, OPAL_ADD_PE_TO_DOMAIN);
+ if (rc)
+ pe_warn(pe, "OPAL error %d adding self to PELTV\n", rc);
opal_pci_eeh_freeze_clear(phb->opal_id, pe->pe_number,
OPAL_EEH_ACTION_CLEAR_FREEZE_ALL);
--
1.7.9.5
^ permalink raw reply related
* [PATCH] powerpc: remove unused REDBOOT Kconfig parameter
From: Michael Opdenacker @ 2013-11-04 8:38 UTC (permalink / raw)
To: benh, paulus, galak, vitb, marcelo
Cc: Michael Opdenacker, linuxppc-dev, linux-kernel
This removes the REDBOOT Kconfig parameter,
which was no longer used anywhere in the source code
and Makefiles.
Signed-off-by: Michael Opdenacker <michael.opdenacker@free-electrons.com>
---
arch/powerpc/Kconfig | 3 ---
arch/powerpc/platforms/83xx/Kconfig | 1 -
arch/powerpc/platforms/8xx/Kconfig | 1 -
3 files changed, 5 deletions(-)
diff --git a/arch/powerpc/Kconfig b/arch/powerpc/Kconfig
index 38f3b7e47ec5..f02a41935c95 100644
--- a/arch/powerpc/Kconfig
+++ b/arch/powerpc/Kconfig
@@ -205,9 +205,6 @@ config DEFAULT_UIMAGE
Used to allow a board to specify it wants a uImage built by default
default n
-config REDBOOT
- bool
-
config ARCH_HIBERNATION_POSSIBLE
bool
default y
diff --git a/arch/powerpc/platforms/83xx/Kconfig b/arch/powerpc/platforms/83xx/Kconfig
index 670a033264c0..2bdc8c862c46 100644
--- a/arch/powerpc/platforms/83xx/Kconfig
+++ b/arch/powerpc/platforms/83xx/Kconfig
@@ -99,7 +99,6 @@ config SBC834x
config ASP834x
bool "Analogue & Micro ASP 834x"
select PPC_MPC834x
- select REDBOOT
help
This enables support for the Analogue & Micro ASP 83xx
board.
diff --git a/arch/powerpc/platforms/8xx/Kconfig b/arch/powerpc/platforms/8xx/Kconfig
index 8dec3c0911ad..bd6f1a1cf922 100644
--- a/arch/powerpc/platforms/8xx/Kconfig
+++ b/arch/powerpc/platforms/8xx/Kconfig
@@ -45,7 +45,6 @@ config PPC_EP88XC
config PPC_ADDER875
bool "Analogue & Micro Adder 875"
select CPM1
- select REDBOOT
help
This enables support for the Analogue & Micro Adder 875
board.
--
1.8.1.2
^ permalink raw reply related
* Re: perf events ring buffer memory barrier on powerpc
From: Peter Zijlstra @ 2013-11-04 9:07 UTC (permalink / raw)
To: Paul E. McKenney
Cc: Michael Neuling, Mathieu Desnoyers, LKML, Oleg Nesterov,
Linux PPC dev, Anton Blanchard, Frederic Weisbecker,
Victor Kaplansky
In-Reply-To: <20131102152048.GI4067@linux.vnet.ibm.com>
On Sat, Nov 02, 2013 at 08:20:48AM -0700, Paul E. McKenney wrote:
> On Fri, Nov 01, 2013 at 11:30:17AM +0100, Peter Zijlstra wrote:
> > Furthermore there's a gazillion parallel userspace programs.
>
> Most of which have very unaggressive concurrency designs.
pthread_mutex_t A, B;
char data_A[x];
int counter_B = 1;
void funA(void)
{
pthread_mutex_lock(&A);
memset(data_A, 0, sizeof(data_A));
pthread_mutex_unlock(&A);
}
void funB(void)
{
pthread_mutex_lock(&B);
counter_B++;
pthread_mutex_unlock(&B);
}
void funC(void)
{
pthread_mutex_lock(&B)
printf("%d\n", counter_B);
pthread_mutex_unlock(&B);
}
Then run: funA, funB, funC concurrently, and end with a funC.
Then explain to userman than his unaggressive program can return:
0
1
Because the memset() thought it might be a cute idea to overwrite
counter_B and fix it up 'later'. Which if I understood you right is
valid in C/C++ :-(
Not that any actual memset implementation exhibiting this trait wouldn't
be shot on the spot.
> > > By marking "ptr" as atomic, thus telling the compiler not to mess with it.
> > > And thus requiring that all accesses to it be decorated, which in the
> > > case of RCU could be buried in the RCU accessors.
> >
> > This seems contradictory; marking it atomic would look like:
> >
> > struct foo {
> > unsigned long value;
> > __atomic void *ptr;
> > unsigned long value1;
> > };
> >
> > Clearly we cannot hide this definition in accessors, because then
> > accesses to value* won't see the annotation.
>
> #define __rcu __atomic
Yeah, except we don't use __rcu all that consistently; in fact I don't
know if I ever added it.
^ permalink raw reply
* Re: perf events ring buffer memory barrier on powerpc
From: Paul E. McKenney @ 2013-11-04 10:00 UTC (permalink / raw)
To: Peter Zijlstra
Cc: Michael Neuling, Mathieu Desnoyers, LKML, Oleg Nesterov,
Linux PPC dev, Anton Blanchard, Frederic Weisbecker,
Victor Kaplansky
In-Reply-To: <20131104090744.GE10651@twins.programming.kicks-ass.net>
On Mon, Nov 04, 2013 at 10:07:44AM +0100, Peter Zijlstra wrote:
> On Sat, Nov 02, 2013 at 08:20:48AM -0700, Paul E. McKenney wrote:
> > On Fri, Nov 01, 2013 at 11:30:17AM +0100, Peter Zijlstra wrote:
> > > Furthermore there's a gazillion parallel userspace programs.
> >
> > Most of which have very unaggressive concurrency designs.
>
> pthread_mutex_t A, B;
>
> char data_A[x];
> int counter_B = 1;
>
> void funA(void)
> {
> pthread_mutex_lock(&A);
> memset(data_A, 0, sizeof(data_A));
> pthread_mutex_unlock(&A);
> }
>
> void funB(void)
> {
> pthread_mutex_lock(&B);
> counter_B++;
> pthread_mutex_unlock(&B);
> }
>
> void funC(void)
> {
> pthread_mutex_lock(&B)
> printf("%d\n", counter_B);
> pthread_mutex_unlock(&B);
> }
>
> Then run: funA, funB, funC concurrently, and end with a funC.
>
> Then explain to userman than his unaggressive program can return:
> 0
> 1
>
> Because the memset() thought it might be a cute idea to overwrite
> counter_B and fix it up 'later'. Which if I understood you right is
> valid in C/C++ :-(
>
> Not that any actual memset implementation exhibiting this trait wouldn't
> be shot on the spot.
Even without such a malicious memcpy() implementation I must still explain
about false sharing when the developer notices that the unaggressive
program isn't running as fast as expected.
> > > > By marking "ptr" as atomic, thus telling the compiler not to mess with it.
> > > > And thus requiring that all accesses to it be decorated, which in the
> > > > case of RCU could be buried in the RCU accessors.
> > >
> > > This seems contradictory; marking it atomic would look like:
> > >
> > > struct foo {
> > > unsigned long value;
> > > __atomic void *ptr;
> > > unsigned long value1;
> > > };
> > >
> > > Clearly we cannot hide this definition in accessors, because then
> > > accesses to value* won't see the annotation.
> >
> > #define __rcu __atomic
>
> Yeah, except we don't use __rcu all that consistently; in fact I don't
> know if I ever added it.
There are more than 300 of them in the kernel. Plus sparse can be
convinced to yell at you if you don't use them. So lack of __rcu could
be fixed without too much trouble.
The C/C++11 need to annotate functions that take arguments or return
values taken from rcu_dereference() is another story. But the compilers
have to get significantly more aggressive or developers have to be doing
unusual things that result in rcu_dereference() returning something whose
value the compiler can predict exactly.
Thanx, Paul
^ permalink raw reply
* RE: [PATCHv2 6/8] ASoC: fsl: add SGTL5000 based audio machine driver.
From: Li Xiubo @ 2013-11-04 9:52 UTC (permalink / raw)
To: Mark Brown
Cc: mark.rutland@arm.com, alsa-devel@alsa-project.org,
linux-doc@vger.kernel.org, tiwai@suse.de, Huan Wang,
timur@tabi.org, perex@perex.cz, Shawn Guo, LW@KARO-electronics.de,
linux@arm.linux.org.uk, Guangyu Chen,
linux-arm-kernel@lists.infradead.org, grant.likely@linaro.org,
devicetree@vger.kernel.org, ian.campbell@citrix.com,
pawel.moll@arm.com, swarren@wwwdotorg.org,
rob.herring@calxeda.com, oskar@scara.com, Fabio Estevam,
lgirdwood@gmail.com, linux-kernel@vger.kernel.org,
rob@landley.net, Zhengxiong Jin, shawn.guo@linaro.org,
linuxppc-dev@lists.ozlabs.org
In-Reply-To: <20131101184008.GF2493@sirena.org.uk>
> > Conflicts:
> > sound/soc/fsl/Makefile
>=20
> Ahem.
>=20
This will be removed.
> > +static int fsl_sgtl5000_remove(struct platform_device *pdev) {
> > + snd_soc_unregister_card(&fsl_sgt1500_card);
> > +
> > + return 0;
> > +}
>=20
> You're using snd_soc_unregister_card() so you don't need to do this.
>
See the next version.
^ permalink raw reply
* Re: [RFC] arch: Introduce new TSO memory barrier smp_tmb()
From: Paul E. McKenney @ 2013-11-04 10:51 UTC (permalink / raw)
To: Linus Torvalds
Cc: Michael Neuling, Mathieu Desnoyers, Peter Zijlstra, Oleg Nesterov,
LKML, Linux PPC dev, Anton Blanchard, Frederic Weisbecker,
Victor Kaplansky
In-Reply-To: <CA+55aFyD_kCkAHQwHCUBrumO-pH6LaZikTNvyWDW_tWsHdqk6Q@mail.gmail.com>
On Sun, Nov 03, 2013 at 03:34:00PM -0800, Linus Torvalds wrote:
> On Sun, Nov 3, 2013 at 2:42 PM, Paul E. McKenney
> <paulmck@linux.vnet.ibm.com> wrote:
> >
> > smp_storebuffer_mb() -- A barrier that enforces those orderings
> > that do not invalidate the hardware store-buffer optimization.
>
> Ugh. Maybe. Can you guarantee that those are the correct semantics?
> And why talk about the hardware semantics, when you really want
> specific semantics for the *software*.
>
> > smp_not_w_r_mb() -- A barrier that orders everything except prior
> > writes against subsequent reads.
>
> Ok, that sounds more along the lines of "these are the semantics we
> want", but I have to say, it also doesn't make me go "ahh, ok".
>
> > smp_acqrel_mb() -- A barrier that combines C/C++ acquire and release
> > semantics. (C/C++ "acquire" orders a specific load against
> > subsequent loads and stores, while C/C++ "release" orders
> > a specific store against prior loads and stores.)
>
> I don't think this is true. acquire+release is much stronger than what
> you're looking for - it doesn't allow subsequent reads to move past
> the write (because that would violate the acquire part). On x86, for
> example, you'd need to have a locked cycle for smp_acqrel_mb().
>
> So again, what are the guarantees you actually want? Describe those.
> And then make a name.
I was thinking in terms of the guarantee that TSO systems provide
given a barrier() directive, and that PowerPC provides given the lwsync
instruction. This guarantee is that loads preceding the barrier will
not be reordered with memory referenced following the barrier, and that
stores preceding the barrier will not be reordered with stores following
the barrier. But given how much easier RCU reviews became after burying
smp_wmb() and smp_read_barrier_depends() into rcu_assign_pointer() and
rcu_dereference(), respectively, I think I prefer an extension of your
idea below.
> I _think_ the guarantees you want is:
> - SMP write barrier
> - *local* read barrier for reads preceding the write.
>
> but the problem is that the "preceding reads" part is really
> specifically about the write that you had. The barrier should really
> be attached to the *particular* write operation, it cannot be a
> standalone barrier.
Indeed, neither rcu_assign_pointer() nor the circular queue really needs a
standalone barrier, so that attaching the barrier to a particular memory
reference would work. And as you note below, in the case of ARM this
would turn into one of their new memory-reference instructions.
> So it would *kind* of act like a "smp_wmb() + smp_rmb()", but the
> problem is that a "smp_rmb()" doesn't really "attach" to the preceding
> write.
>
> This is analogous to a "acquire" operation: you cannot make an
> "acquire" barrier, because it's not a barrier *between* two ops, it's
> associated with one particular op.
But you -could- use any barrier that prevented reordering of any preceding
load with any subsequent memory reference. Please note that I am -not-
advocating this anymore, because I like the idea of attaching the barrier
to a particular memory operation. However, for completeness, here it is
in the case of TSO systems and PowerPC, respectively:
#define smp_acquire_mb() barrier();
#define smp_acquire_mb() \
__asm__ __volatile__ (stringify_in_c(LWSYNC) : : :"memory");
This functions correctly, but is a pain to review because you have to
figure out which of many possible preceding loads the smp_acquire_mb()
is supposed to attach to. As you say, it is -way- better to attach the
barrier to a particular memory operation.
> So what I *think* you actually really really want is a "store with
> release consistency, followed by a write barrier".
I believe that the combination of "store with release consistency" and
"load with acquire consistency" should do the trick for the two use cases
at this point, which again are circular buffers and rcu_assign_pointer().
At this point, I don't see the need for "followed by a write barrier".
But I step through the circular buffers below.
> In TSO, afaik all stores have release consistency, and all writes are
> ordered, which is why this is a no-op in TSO. And x86 also has that
> "all stores have release consistency, and all writes are ordered"
> model, even if TSO doesn't really describe the x86 model.
Yep, as does the mainframe. And these architectures also have all reads
having acquire consistency.
> But on ARM64, for example, I think you'd really want the store itself
> to be done with "stlr" (store with release), and then follow up with a
> "dsb st" after that.
Agree with the "stlr" but don't (yet, anyway) understand the need for
a subsequent "dsb st".
> And notice how that requires you to mark the store itself. There is no
> actual barrier *after* the store that does the optimized model.
And marking the store itself is a very good thing from my viewpoint.
> Of course, it's entirely possible that it's not worth worrying about
> this on ARM64, and that just doing it as a "normal store followed by a
> full memory barrier" is good enough. But at least in *theory* a
> microarchitecture might make it much cheaper to do a "store with
> release consistency" followed by "write barrier".
>
> Anyway, having talked exhaustively about exactly what semantics you
> are after, I *think* the best model would be to just have a
>
> #define smp_store_with_release_semantics(x, y) ...
>
> and use that *and* a "smp_wmb()" for this (possibly a special
> "smp_wmb_after_release()" if that allows people to avoid double
> barriers). On x86 (and TSO systems), the
> smp_store_with_release_semantics() would be just a regular store, and
> the smp_wmb() is obviously a no-op. Other platforms would end up doing
> other things.
>
> Hmm?
OK, something like this for the definitions (though PowerPC might want
to locally abstract the lwsync expansion):
#define smp_store_with_release_semantics(p, v) /* x86, s390, etc. */ \
do { \
barrier(); \
ACCESS_ONCE(p) = (v); \
} while (0)
#define smp_store_with_release_semantics(p, v) /* PowerPC. */ \
do { \
__asm__ __volatile__ (stringify_in_c(LWSYNC) : : :"memory"); \
ACCESS_ONCE(p) = (v); \
} while (0)
#define smp_load_with_acquire_semantics(p) /* x86, s390, etc. */ \
({ \
typeof(*p) *_________p1 = ACCESS_ONCE(p); \
barrier(); \
_________p1; \
})
#define smp_load_with_acquire_semantics(p) /* PowerPC. */ \
({ \
typeof(*p) *_________p1 = ACCESS_ONCE(p); \
__asm__ __volatile__ (stringify_in_c(LWSYNC) : : :"memory"); \
_________p1; \
})
For ARM, smp_load_with_acquire_semantics() is a wrapper around the ARM
"ldar" instruction and smp_store_with_release_semantics() is a wrapper
around the ARM "stlr" instruction.
Then if I am not too confused (and I would expect Victor to let me know
in short order if I am), the following patch to the current mainline
version of Documentation/circular-buffers.txt would suffice.
Thoughts?
Thanx, Paul
------------------------------------------------------------------------
diff --git a/Documentation/circular-buffers.txt b/Documentation/circular-buffers.txt
index 8117e5bf6065..1846044bf6cc 100644
--- a/Documentation/circular-buffers.txt
+++ b/Documentation/circular-buffers.txt
@@ -160,6 +160,7 @@ The producer will look something like this:
spin_lock(&producer_lock);
unsigned long head = buffer->head;
+ /* The spin_unlock() and next spin_lock() provide needed ordering. */
unsigned long tail = ACCESS_ONCE(buffer->tail);
if (CIRC_SPACE(head, tail, buffer->size) >= 1) {
@@ -168,9 +169,8 @@ The producer will look something like this:
produce_item(item);
- smp_wmb(); /* commit the item before incrementing the head */
-
- buffer->head = (head + 1) & (buffer->size - 1);
+ smp_store_with_release_semantics(buffer->head,
+ (head + 1) & (buffer->size - 1));
/* wake_up() will make sure that the head is committed before
* waking anyone up */
@@ -183,9 +183,14 @@ This will instruct the CPU that the contents of the new item must be written
before the head index makes it available to the consumer and then instructs the
CPU that the revised head index must be written before the consumer is woken.
-Note that wake_up() doesn't have to be the exact mechanism used, but whatever
-is used must guarantee a (write) memory barrier between the update of the head
-index and the change of state of the consumer, if a change of state occurs.
+Note that wake_up() does not guarantee any sort of barrier unless something
+is actually awakened. We therefore cannot rely on it for ordering. However,
+there is always one element of the array left empty. Therefore, the
+producer must produce two elements before it could possibly corrupt the
+element currently being read by the consumer. Therefore, the unlock-lock
+pair between consecutive invocations of the consumer provides the necessary
+ordering between the read of the index indicating that the consumer has
+vacated a given element and the write by the producer to that same element.
THE CONSUMER
@@ -195,21 +200,18 @@ The consumer will look something like this:
spin_lock(&consumer_lock);
- unsigned long head = ACCESS_ONCE(buffer->head);
+ unsigned long head = smp_load_with_acquire_semantics(buffer->head);
unsigned long tail = buffer->tail;
if (CIRC_CNT(head, tail, buffer->size) >= 1) {
- /* read index before reading contents at that index */
- smp_read_barrier_depends();
/* extract one item from the buffer */
struct item *item = buffer[tail];
consume_item(item);
- smp_mb(); /* finish reading descriptor before incrementing tail */
-
- buffer->tail = (tail + 1) & (buffer->size - 1);
+ smp_store_with_release_semantics(buffer->tail,
+ (tail + 1) & (buffer->size - 1));
}
spin_unlock(&consumer_lock);
@@ -218,12 +220,17 @@ This will instruct the CPU to make sure the index is up to date before reading
the new item, and then it shall make sure the CPU has finished reading the item
before it writes the new tail pointer, which will erase the item.
-
-Note the use of ACCESS_ONCE() in both algorithms to read the opposition index.
-This prevents the compiler from discarding and reloading its cached value -
-which some compilers will do across smp_read_barrier_depends(). This isn't
-strictly needed if you can be sure that the opposition index will _only_ be
-used the once.
+Note the use of ACCESS_ONCE() and smp_load_with_acquire_semantics()
+to read the opposition index. This prevents the compiler from
+discarding and reloading its cached value - which some compilers will
+do across smp_read_barrier_depends(). This isn't strictly needed
+if you can be sure that the opposition index will _only_ be used
+the once. The smp_load_with_acquire_semantics() additionally forces
+the CPU to order against subsequent memory references. Similarly,
+smp_store_with_release_semantics() is used in both algorithms to write
+the thread's index. This documents the fact that we are writing to
+something that can be read concurrently, prevents the compiler from
+tearing the store, and enforces ordering against previous accesses.
===============
^ permalink raw reply related
* Re: [RFC] arch: Introduce new TSO memory barrier smp_tmb()
From: Will Deacon @ 2013-11-04 11:05 UTC (permalink / raw)
To: Linus Torvalds
Cc: Michael Neuling, Mathieu Desnoyers, Peter Zijlstra, Oleg Nesterov,
LKML, Linux PPC dev, Anton Blanchard, Frederic Weisbecker,
Victor Kaplansky, Paul McKenney
In-Reply-To: <CA+55aFyD_kCkAHQwHCUBrumO-pH6LaZikTNvyWDW_tWsHdqk6Q@mail.gmail.com>
On Sun, Nov 03, 2013 at 11:34:00PM +0000, Linus Torvalds wrote:
> So it would *kind* of act like a "smp_wmb() + smp_rmb()", but the
> problem is that a "smp_rmb()" doesn't really "attach" to the preceding
> write.
Agreed.
> This is analogous to a "acquire" operation: you cannot make an
> "acquire" barrier, because it's not a barrier *between* two ops, it's
> associated with one particular op.
>
> So what I *think* you actually really really want is a "store with
> release consistency, followed by a write barrier".
How does that order reads against reads? (Paul mentioned this as a
requirement). I not clear about the use case for this, so perhaps there is a
dependency that I'm not aware of.
> In TSO, afaik all stores have release consistency, and all writes are
> ordered, which is why this is a no-op in TSO. And x86 also has that
> "all stores have release consistency, and all writes are ordered"
> model, even if TSO doesn't really describe the x86 model.
>
> But on ARM64, for example, I think you'd really want the store itself
> to be done with "stlr" (store with release), and then follow up with a
> "dsb st" after that.
So a dsb is pretty heavyweight here (it prevents execution of *any* further
instructions until all preceeding stores have completed, as well as
ensuring completion of any ongoing cache flushes). In conjunction with the
store-release, that's going to hold everything up until the store-release
(and therefore any preceeding memory accesses) have completed. Granted, I
think that gives Paul his read/read ordering, but it's a lot heavier than
what's required.
> And notice how that requires you to mark the store itself. There is no
> actual barrier *after* the store that does the optimized model.
>
> Of course, it's entirely possible that it's not worth worrying about
> this on ARM64, and that just doing it as a "normal store followed by a
> full memory barrier" is good enough. But at least in *theory* a
> microarchitecture might make it much cheaper to do a "store with
> release consistency" followed by "write barrier".
I agree with the sentiment but, given that this stuff is so heavily
microarchitecture-dependent (and not simple to probe), a simple dmb ish
might be the best option after all. That's especially true if the
microarchitecture decided to ignore the barrier options and treat everything
as `all accesses, full system' in order to keep the hardware design simple.
Will
^ permalink raw reply
* Re: [RFC] arch: Introduce new TSO memory barrier smp_tmb()
From: Peter Zijlstra @ 2013-11-04 11:22 UTC (permalink / raw)
To: Paul E. McKenney
Cc: Michael Neuling, Mathieu Desnoyers, Oleg Nesterov, LKML,
Linux PPC dev, Anton Blanchard, Frederic Weisbecker,
Victor Kaplansky, Linus Torvalds
In-Reply-To: <20131104105059.GL3947@linux.vnet.ibm.com>
On Mon, Nov 04, 2013 at 02:51:00AM -0800, Paul E. McKenney wrote:
> OK, something like this for the definitions (though PowerPC might want
> to locally abstract the lwsync expansion):
>
> #define smp_store_with_release_semantics(p, v) /* x86, s390, etc. */ \
> do { \
> barrier(); \
> ACCESS_ONCE(p) = (v); \
> } while (0)
>
> #define smp_store_with_release_semantics(p, v) /* PowerPC. */ \
> do { \
> __asm__ __volatile__ (stringify_in_c(LWSYNC) : : :"memory"); \
> ACCESS_ONCE(p) = (v); \
> } while (0)
>
> #define smp_load_with_acquire_semantics(p) /* x86, s390, etc. */ \
> ({ \
> typeof(*p) *_________p1 = ACCESS_ONCE(p); \
> barrier(); \
> _________p1; \
> })
>
> #define smp_load_with_acquire_semantics(p) /* PowerPC. */ \
> ({ \
> typeof(*p) *_________p1 = ACCESS_ONCE(p); \
> __asm__ __volatile__ (stringify_in_c(LWSYNC) : : :"memory"); \
> _________p1; \
> })
>
> For ARM, smp_load_with_acquire_semantics() is a wrapper around the ARM
> "ldar" instruction and smp_store_with_release_semantics() is a wrapper
> around the ARM "stlr" instruction.
This still leaves me confused as to what to do with my case :/
Slightly modified since last time -- as the simplified version was maybe
simplified too far.
To recap, I'd like to get rid of barrier A where possible, since that's
now a full barrier for every event written.
However, there's no immediate store I can attach it to; the obvious one
would be the kbuf->head store, but that's complicated by the
local_cmpxchg() thing.
And we need that cmpxchg loop because a hardware NMI event can
interleave with a software event.
And to be honest, I'm still totally confused about memory barriers vs
control flow vs C/C++. The only way we're ever getting to that memcpy is
if we've already observed ubuf->tail, so that LOAD has to be fully
processes and completed.
I'm really not seeing how a STORE from the memcpy() could possibly go
wrong; and if C/C++ can hoist the memcpy() over a compiler barrier()
then I suppose we should all just go home.
/me who wants A to be a barrier() but is terminally confused.
---
/*
* One important detail is that the kbuf part and the kbuf_writer() are
* strictly per cpu and we can thus rely on program order for those.
*
* Only the userspace consumer can possibly run on another cpu, and thus we
* need to ensure data consistency for those.
*/
struct buffer {
u64 size;
u64 tail;
u64 head;
void *data;
};
struct buffer *kbuf, *ubuf;
/*
* If there's space in the buffer; store the data @buf; otherwise
* discard it.
*/
void kbuf_write(int sz, void *buf)
{
u64 tail, head, offset;
do {
tail = ACCESS_ONCE(ubuf->tail);
offset = head = kbuf->head;
if (CIRC_SPACE(head, tail, kbuf->size) < sz) {
/* discard @buf */
return;
}
head += sz;
} while (local_cmpxchg(&kbuf->head, offset, head) != offset)
/*
* Ensure that if we see the userspace tail (ubuf->tail) such
* that there is space to write @buf without overwriting data
* userspace hasn't seen yet, we won't in fact store data before
* that read completes.
*/
smp_mb(); /* A, matches with D */
memcpy(kbuf->data + offset, buf, sz);
/*
* Ensure that we write all the @buf data before we update the
* userspace visible ubuf->head pointer.
*/
smp_wmb(); /* B, matches with C */
ubuf->head = kbuf->head;
}
/*
* Consume the buffer data and update the tail pointer to indicate to
* kernel space there's 'free' space.
*/
void ubuf_read(void)
{
u64 head, tail;
tail = ACCESS_ONCE(ubuf->tail);
head = ACCESS_ONCE(ubuf->head);
/*
* Ensure we read the buffer boundaries before the actual buffer
* data...
*/
smp_rmb(); /* C, matches with B */
while (tail != head) {
obj = ubuf->data + tail;
/* process obj */
tail += obj->size;
tail %= ubuf->size;
}
/*
* Ensure all data reads are complete before we issue the
* ubuf->tail update; once that update hits, kbuf_write() can
* observe and overwrite data.
*/
smp_mb(); /* D, matches with A */
ubuf->tail = tail;
}
^ permalink raw reply
* Re: [V2 PATCH 3/3] powerpc: Fix Unaligned LE Floating Point Loads and Stores
From: Tom Musta @ 2013-11-04 13:29 UTC (permalink / raw)
To: Benjamin Herrenschmidt; +Cc: linuxppc-dev
In-Reply-To: <1383532491.4776.42.camel@pasglop>
On 11/3/2013 8:34 PM, Benjamin Herrenschmidt wrote:
> On Thu, 2013-10-31 at 13:38 -0500, Tom wrote:
>> From: Tom Musta <tommusta@gmail.com>
>>
>> This patch addresses unaligned single precision floating point loads
>> and stores in the single-step code. The old implementation
>> improperly treated an 8 byte structure as an array of two 4 byte
>> words, which is a classic little endian bug.
>
> Do that patch differ from v1 ? I also already merged v1 of this
> one (the only one I didn't merge is the emulate_step one)
>
> Cheers,
> Ben.
>
Ben: Only patch 1/3 (Enable emulate_step in Little Endian Mode) differs in V2.
^ permalink raw reply
* Re: [PATCHv2 1/8] ALSA: Add SAI SoC Digital Audio Interface driver.
From: Mark Brown @ 2013-11-04 16:15 UTC (permalink / raw)
To: Li Xiubo
Cc: mark.rutland@arm.com, alsa-devel@alsa-project.org,
linux-doc@vger.kernel.org, tiwai@suse.de, Huan Wang,
timur@tabi.org, perex@perex.cz, Shawn Guo, LW@KARO-electronics.de,
linux@arm.linux.org.uk, Guangyu Chen,
linux-arm-kernel@lists.infradead.org, grant.likely@linaro.org,
devicetree@vger.kernel.org, ian.campbell@citrix.com,
pawel.moll@arm.com, swarren@wwwdotorg.org,
rob.herring@calxeda.com, oskar@scara.com, Fabio Estevam,
lgirdwood@gmail.com, linux-kernel@vger.kernel.org,
rob@landley.net, Zhengxiong Jin, shawn.guo@linaro.org,
linuxppc-dev@lists.ozlabs.org
In-Reply-To: <1DD289F6464F0949A2FCA5AA6DC23F8287439D@039-SN2MPN1-013.039d.mgd.msft.net>
[-- Attachment #1: Type: text/plain, Size: 2209 bytes --]
On Mon, Nov 04, 2013 at 07:35:12AM +0000, Li Xiubo wrote:
> From the ASoC subsystem comments we can see that:
> ++++++
> Configures the clock dividers. This is used to derive the best DAI bit and
> frame clocks from the system or master clock. It's best to set the DAI bit
> and frame clocks as low as possible to save system power.
> ------
You should never use this unless you have to, there is no point in every
single machine driver using your driver having to duplicate the same
calculations.
>
> > > +static int fsl_sai_dai_probe(struct snd_soc_dai *dai) {
> > > + int ret;
> > > + struct fsl_sai *sai = dev_get_drvdata(dai->dev);
> > > +
> > > + ret = clk_prepare_enable(sai->clk);
> > > + if (ret)
> > > + return ret;
> > It'd be nicer to only enable the clock while the device is in active use.
> While if the module clock is not enabled here, the followed registers cannot read/write in the same function.
> And this _probe function is the _dai_probe not the driver's module _probe.
So you can enable the clock when you explicitly need to write to the
registers...
> If the clk_prepare_enable(sai->clk) is not here, where should it be will be nicer ?
> One of the following functions ?
> .set_sysclk = fsl_sai_set_dai_sysclk,
> .set_clkdiv = fsl_sai_set_dai_clkdiv,
> .set_fmt = fsl_sai_set_dai_fmt,
> .set_tdm_slot = fsl_sai_set_dai_tdm_slot,
> .hw_params = fsl_sai_hw_params,
> .trigger = fsl_sai_trigger,
It could be in any or all of them except trigger (where the core should
hold a runtime PM reference anyway). You can always take a reference
for the duration of the function if you're concerned it may be called
when the referent isn't otherwise held.
> > > + ret = snd_dmaengine_pcm_register(&pdev->dev, NULL,
> > > + SND_DMAENGINE_PCM_FLAG_NO_RESIDUE);
> > > + if (ret)
> > > + return ret;
> > We should have a devm_ version of this.
> Sorry, is there one patch for adding the devm_ version of snd_dmaengine_pcm_register() already ?
> In the -next and other topics branches I could not find it.
No, there isn't one but there should be one.
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]
^ permalink raw reply
* Re: [RFC] arch: Introduce new TSO memory barrier smp_tmb()
From: Paul E. McKenney @ 2013-11-04 16:27 UTC (permalink / raw)
To: Peter Zijlstra
Cc: Michael Neuling, Mathieu Desnoyers, Oleg Nesterov, LKML,
Linux PPC dev, Anton Blanchard, Frederic Weisbecker,
Victor Kaplansky, Linus Torvalds
In-Reply-To: <20131104112254.GK28601@twins.programming.kicks-ass.net>
On Mon, Nov 04, 2013 at 12:22:54PM +0100, Peter Zijlstra wrote:
> On Mon, Nov 04, 2013 at 02:51:00AM -0800, Paul E. McKenney wrote:
> > OK, something like this for the definitions (though PowerPC might want
> > to locally abstract the lwsync expansion):
> >
> > #define smp_store_with_release_semantics(p, v) /* x86, s390, etc. */ \
> > do { \
> > barrier(); \
> > ACCESS_ONCE(p) = (v); \
> > } while (0)
> >
> > #define smp_store_with_release_semantics(p, v) /* PowerPC. */ \
> > do { \
> > __asm__ __volatile__ (stringify_in_c(LWSYNC) : : :"memory"); \
> > ACCESS_ONCE(p) = (v); \
> > } while (0)
> >
> > #define smp_load_with_acquire_semantics(p) /* x86, s390, etc. */ \
> > ({ \
> > typeof(*p) *_________p1 = ACCESS_ONCE(p); \
> > barrier(); \
> > _________p1; \
> > })
> >
> > #define smp_load_with_acquire_semantics(p) /* PowerPC. */ \
> > ({ \
> > typeof(*p) *_________p1 = ACCESS_ONCE(p); \
> > __asm__ __volatile__ (stringify_in_c(LWSYNC) : : :"memory"); \
> > _________p1; \
> > })
> >
> > For ARM, smp_load_with_acquire_semantics() is a wrapper around the ARM
> > "ldar" instruction and smp_store_with_release_semantics() is a wrapper
> > around the ARM "stlr" instruction.
>
> This still leaves me confused as to what to do with my case :/
>
> Slightly modified since last time -- as the simplified version was maybe
> simplified too far.
>
> To recap, I'd like to get rid of barrier A where possible, since that's
> now a full barrier for every event written.
>
> However, there's no immediate store I can attach it to; the obvious one
> would be the kbuf->head store, but that's complicated by the
> local_cmpxchg() thing.
>
> And we need that cmpxchg loop because a hardware NMI event can
> interleave with a software event.
>
> And to be honest, I'm still totally confused about memory barriers vs
> control flow vs C/C++. The only way we're ever getting to that memcpy is
> if we've already observed ubuf->tail, so that LOAD has to be fully
> processes and completed.
>
> I'm really not seeing how a STORE from the memcpy() could possibly go
> wrong; and if C/C++ can hoist the memcpy() over a compiler barrier()
> then I suppose we should all just go home.
>
> /me who wants A to be a barrier() but is terminally confused.
Well, let's see...
> ---
>
>
> /*
> * One important detail is that the kbuf part and the kbuf_writer() are
> * strictly per cpu and we can thus rely on program order for those.
> *
> * Only the userspace consumer can possibly run on another cpu, and thus we
> * need to ensure data consistency for those.
> */
>
> struct buffer {
> u64 size;
> u64 tail;
> u64 head;
> void *data;
> };
>
> struct buffer *kbuf, *ubuf;
>
> /*
> * If there's space in the buffer; store the data @buf; otherwise
> * discard it.
> */
> void kbuf_write(int sz, void *buf)
> {
> u64 tail, head, offset;
>
> do {
> tail = ACCESS_ONCE(ubuf->tail);
So the above load is the key load. It determines whether or not we
have space in the buffer. This of course assumes that only this CPU
writes to ->head.
If so, then:
tail = smp_load_with_acquire_semantics(ubuf->tail); /* A -> D */
> offset = head = kbuf->head;
> if (CIRC_SPACE(head, tail, kbuf->size) < sz) {
> /* discard @buf */
> return;
> }
> head += sz;
> } while (local_cmpxchg(&kbuf->head, offset, head) != offset)
If there is an issue with kbuf->head, presumably local_cmpxchg() fails
and we retry.
But sheesh, do you think we could have buried the definitions of
local_cmpxchg() under a few more layers of macro expansion just to
keep things more obscure? Anyway, griping aside...
o __cmpxchg_local_generic() in include/asm-generic/cmpxchg-local.h
doesn't seem to exclude NMIs, so is not safe for this usage.
o __cmpxchg_local() in ARM handles NMI as long as the
argument is 32 bits, otherwise, it uses the aforementionted
__cmpxchg_local_generic(), which does not handle NMI. Given your
u64, this does not look good...
And some ARM subarches (e.g., metag) seem to fail to handle NMI
even in the 32-bit case.
o FRV and M32r seem to act similar to ARM.
Or maybe these architectures don't do NMIs? If they do, local_cmpxchg()
does not seem to be safe against NMIs in general. :-/
That said, powerpc, 64-bit s390, sparc, and x86 seem to handle it.
Of course, x86's local_cmpxchg() has full memory barriers implicitly.
>
> /*
> * Ensure that if we see the userspace tail (ubuf->tail) such
> * that there is space to write @buf without overwriting data
> * userspace hasn't seen yet, we won't in fact store data before
> * that read completes.
> */
>
> smp_mb(); /* A, matches with D */
Given a change to smp_load_with_acquire_semantics() above, you should not
need this smp_mb().
> memcpy(kbuf->data + offset, buf, sz);
>
> /*
> * Ensure that we write all the @buf data before we update the
> * userspace visible ubuf->head pointer.
> */
> smp_wmb(); /* B, matches with C */
>
> ubuf->head = kbuf->head;
Replace the smp_wmb() and the assignment with:
smp_store_with_release_semantics(ubuf->head, kbuf->head); /* B -> C */
> }
>
> /*
> * Consume the buffer data and update the tail pointer to indicate to
> * kernel space there's 'free' space.
> */
> void ubuf_read(void)
> {
> u64 head, tail;
>
> tail = ACCESS_ONCE(ubuf->tail);
Does anyone else write tail? Or is this defense against NMIs?
If no one else writes to tail and if NMIs cannot muck things up, then
the above ACCESS_ONCE() is not needed, though I would not object to its
staying.
> head = ACCESS_ONCE(ubuf->head);
Make the above be:
head = smp_load_with_acquire_semantics(ubuf->head); /* C -> B */
> /*
> * Ensure we read the buffer boundaries before the actual buffer
> * data...
> */
> smp_rmb(); /* C, matches with B */
And drop the above memory barrier.
> while (tail != head) {
> obj = ubuf->data + tail;
> /* process obj */
> tail += obj->size;
> tail %= ubuf->size;
> }
>
> /*
> * Ensure all data reads are complete before we issue the
> * ubuf->tail update; once that update hits, kbuf_write() can
> * observe and overwrite data.
> */
> smp_mb(); /* D, matches with A */
>
> ubuf->tail = tail;
Replace the above barrier and the assignment with:
smp_store_with_release_semantics(ubuf->tail, tail); /* D -> B. */
> }
All this is leading me to suggest the following shortenings of names:
smp_load_with_acquire_semantics() -> smp_load_acquire()
smp_store_with_release_semantics() -> smp_store_release()
But names aside, the above gets rid of explicit barriers on TSO architectures,
allows ARM to avoid full DMB, and allows PowerPC to use lwsync instead of
the heavier-weight sync.
Thanx, Paul
^ permalink raw reply
* Re: [RFC] arch: Introduce new TSO memory barrier smp_tmb()
From: Peter Zijlstra @ 2013-11-04 16:48 UTC (permalink / raw)
To: Paul E. McKenney
Cc: Michael Neuling, Mathieu Desnoyers, Oleg Nesterov, LKML,
Linux PPC dev, Anton Blanchard, Frederic Weisbecker,
Victor Kaplansky, Linus Torvalds
In-Reply-To: <20131104162732.GN3947@linux.vnet.ibm.com>
On Mon, Nov 04, 2013 at 08:27:32AM -0800, Paul E. McKenney wrote:
> >
> >
> > /*
> > * One important detail is that the kbuf part and the kbuf_writer() are
> > * strictly per cpu and we can thus rely on program order for those.
> > *
> > * Only the userspace consumer can possibly run on another cpu, and thus we
> > * need to ensure data consistency for those.
> > */
> >
> > struct buffer {
> > u64 size;
> > u64 tail;
> > u64 head;
> > void *data;
> > };
> >
> > struct buffer *kbuf, *ubuf;
> >
> > /*
> > * If there's space in the buffer; store the data @buf; otherwise
> > * discard it.
> > */
> > void kbuf_write(int sz, void *buf)
> > {
> > u64 tail, head, offset;
> >
> > do {
> > tail = ACCESS_ONCE(ubuf->tail);
>
> So the above load is the key load. It determines whether or not we
> have space in the buffer. This of course assumes that only this CPU
> writes to ->head.
This assumption is true.
> If so, then:
>
> tail = smp_load_with_acquire_semantics(ubuf->tail); /* A -> D */
>
OK, the way I understand ACQUIRE semantics are the semi-permeable LOCK
semantics from Documetnation/memory-barriers.txt. In which case the
relevant STORES below could be hoisted up here, but not across the READ,
which I suppose is sufficient.
> > offset = head = kbuf->head;
> > if (CIRC_SPACE(head, tail, kbuf->size) < sz) {
> > /* discard @buf */
> > return;
> > }
> > head += sz;
> > } while (local_cmpxchg(&kbuf->head, offset, head) != offset)
>
> If there is an issue with kbuf->head, presumably local_cmpxchg() fails
> and we retry.
>
> But sheesh, do you think we could have buried the definitions of
> local_cmpxchg() under a few more layers of macro expansion just to
> keep things more obscure? Anyway, griping aside...
>
> o __cmpxchg_local_generic() in include/asm-generic/cmpxchg-local.h
> doesn't seem to exclude NMIs, so is not safe for this usage.
>
> o __cmpxchg_local() in ARM handles NMI as long as the
> argument is 32 bits, otherwise, it uses the aforementionted
> __cmpxchg_local_generic(), which does not handle NMI. Given your
> u64, this does not look good...
>
> And some ARM subarches (e.g., metag) seem to fail to handle NMI
> even in the 32-bit case.
>
> o FRV and M32r seem to act similar to ARM.
>
> Or maybe these architectures don't do NMIs? If they do, local_cmpxchg()
> does not seem to be safe against NMIs in general. :-/
>
> That said, powerpc, 64-bit s390, sparc, and x86 seem to handle it.
Ah my bad, so the in-kernel kbuf variant uses unsigned long, which on
all archs should be the native words size and cover the address space.
Only the public variant (ubuf) is u64 wide to not change data structure
layout on compat etc.
I suppose this was a victim on the simplification :/
And in case of 32bit the upper word will always be zero and the partial
reads should all work out just fine.
> Of course, x86's local_cmpxchg() has full memory barriers implicitly.
Not quite, the 'lock' in __raw_cmpxchg() expands to "" due to
__cmpxchg_local(), etc..
> >
> > /*
> > * Ensure that if we see the userspace tail (ubuf->tail) such
> > * that there is space to write @buf without overwriting data
> > * userspace hasn't seen yet, we won't in fact store data before
> > * that read completes.
> > */
> >
> > smp_mb(); /* A, matches with D */
>
> Given a change to smp_load_with_acquire_semantics() above, you should not
> need this smp_mb().
Because the STORES can not be hoisted across the ACQUIRE, indeed.
>
> > memcpy(kbuf->data + offset, buf, sz);
> >
> > /*
> > * Ensure that we write all the @buf data before we update the
> > * userspace visible ubuf->head pointer.
> > */
> > smp_wmb(); /* B, matches with C */
> >
> > ubuf->head = kbuf->head;
>
> Replace the smp_wmb() and the assignment with:
>
> smp_store_with_release_semantics(ubuf->head, kbuf->head); /* B -> C */
And here the RELEASE semantics I assume are the same as the
semi-permeable UNLOCK from _The_ document? In which case the above
STORES cannot be lowered across this store and all should, again, be
well.
> > }
> >
> > /*
> > * Consume the buffer data and update the tail pointer to indicate to
> > * kernel space there's 'free' space.
> > */
> > void ubuf_read(void)
> > {
> > u64 head, tail;
> >
> > tail = ACCESS_ONCE(ubuf->tail);
>
> Does anyone else write tail? Or is this defense against NMIs?
No, we're the sole writer; just general paranoia. Not sure the actual
userspace does this; /me checks. Nope, distinct lack of ACCESS_ONCE()
there, just the rmb(), which including a barrier() should hopefully
accomplish similar things most of the time ;-)
I'll need to introduce ACCESS_ONCE to the userspace code.
> If no one else writes to tail and if NMIs cannot muck things up, then
> the above ACCESS_ONCE() is not needed, though I would not object to its
> staying.
>
> > head = ACCESS_ONCE(ubuf->head);
>
> Make the above be:
>
> head = smp_load_with_acquire_semantics(ubuf->head); /* C -> B */
>
> > /*
> > * Ensure we read the buffer boundaries before the actual buffer
> > * data...
> > */
> > smp_rmb(); /* C, matches with B */
>
> And drop the above memory barrier.
>
> > while (tail != head) {
> > obj = ubuf->data + tail;
> > /* process obj */
> > tail += obj->size;
> > tail %= ubuf->size;
> > }
> >
> > /*
> > * Ensure all data reads are complete before we issue the
> > * ubuf->tail update; once that update hits, kbuf_write() can
> > * observe and overwrite data.
> > */
> > smp_mb(); /* D, matches with A */
> >
> > ubuf->tail = tail;
>
> Replace the above barrier and the assignment with:
>
> smp_store_with_release_semantics(ubuf->tail, tail); /* D -> B. */
>
> > }
Right, so this consumer side isn't called that often and the two
barriers are only per consume, not per event, so I don't care too much
about these.
It would also mean hoisting the implementation of the proposed
primitives into userspace -- which reminds me: should we make
include/asm/barrier.h a uapi header?
> All this is leading me to suggest the following shortenings of names:
>
> smp_load_with_acquire_semantics() -> smp_load_acquire()
>
> smp_store_with_release_semantics() -> smp_store_release()
>
> But names aside, the above gets rid of explicit barriers on TSO architectures,
> allows ARM to avoid full DMB, and allows PowerPC to use lwsync instead of
> the heavier-weight sync.
Totally awesome! ;-) And full ack on the shorter names.
^ permalink raw reply
* [PATCH 0/6] powerpc/math-emu: e500 SPE float emulation fixes
From: Joseph S. Myers @ 2013-11-04 16:50 UTC (permalink / raw)
To: linuxppc-dev; +Cc: Liu Yu, linux-kernel, Shan Hai
This patch series fixes various problems with the floating-point
emulation code for powerpc e500 SPE (some being issues with the
e500-specific emulation code, some with the generic math-emu headers).
All six patches were sent individually last month as the issues were
identified and fixed in the course of preparing the e500 glibc port,
and received no comments. There are no substantive changes to the
patches in this version, but I've retested the glibc port (which is
now upstream, along with all the generic math-emu changes relevant to
the glibc soft-fp code, and various fixes to soft-fp corresponding to
fixes in the kernel code in the hope that at some point we can get the
kernel using the current soft-fp code again) with current kernel
sources with this patch series applied.
The only dependencies between patches in this series should be that
patch 5 (fix e500 SPE float to integer and fixed-point conversions)
depends on patch 2 (fix e500 SPE float rounding inexactness
detection). Other than that, I think any subset of the patches can be
applied in any order, if some subset seems OK but there are concerns
about other patches in the series.
--
Joseph S. Myers
joseph@codesourcery.com
^ permalink raw reply
* [PATCH 1/6] powerpc: fix exception clearing in e500 SPE float emulation
From: Joseph S. Myers @ 2013-11-04 16:52 UTC (permalink / raw)
To: linuxppc-dev; +Cc: Liu Yu, linux-kernel, Shan Hai
In-Reply-To: <Pine.LNX.4.64.1311041649250.4290@digraph.polyomino.org.uk>
From: Joseph Myers <joseph@codesourcery.com>
The e500 SPE floating-point emulation code clears existing exceptions
(__FPU_FPSCR &= ~FP_EX_MASK;) before ORing in the exceptions from the
emulated operation. However, these exception bits are the "sticky",
cumulative exception bits, and should only be cleared by the user
program setting SPEFSCR, not implicitly by any floating-point
instruction (whether executed purely by the hardware or emulated).
The spurious clearing of these bits shows up as missing exceptions in
glibc testing.
Fixing this, however, is not as simple as just not clearing the bits,
because while the bits may be from previous floating-point operations
(in which case they should not be cleared), the processor can also set
the sticky bits itself before the interrupt for an exception occurs,
and this can happen in cases when IEEE 754 semantics are that the
sticky bit should not be set. Specifically, the "invalid" sticky bit
is set in various cases with non-finite operands, where IEEE 754
semantics do not involve raising such an exception, and the
"underflow" sticky bit is set in cases of exact underflow, whereas
IEEE 754 semantics are that this flag is set only for inexact
underflow. Thus, for correct emulation the kernel needs to know the
setting of these two sticky bits before the instruction being
emulated.
When a floating-point operation raises an exception, the kernel can
note the state of the sticky bits immediately afterwards. Some
<fenv.h> functions that affect the state of these bits, such as
fesetenv and feholdexcept, need to use prctl with PR_GET_FPEXC and
PR_SET_FPEXC anyway, and so it is natural to record the state of those
bits during that call into the kernel and so avoid any need for a
separate call into the kernel to inform it of a change to those bits.
Thus, the interface I chose to use (in this patch and the glibc port)
is that one of those prctl calls must be made after any userspace
change to those sticky bits, other than through a floating-point
operation that traps into the kernel anyway. feclearexcept and
fesetexceptflag duly make those calls, which would not be required
were it not for this issue.
Signed-off-by: Joseph Myers <joseph@codesourcery.com>
---
Previous submission: <http://lkml.org/lkml/2013/10/4/495>.
diff --git a/arch/powerpc/include/asm/processor.h b/arch/powerpc/include/asm/processor.h
index ce4de5a..0b02e23 100644
--- a/arch/powerpc/include/asm/processor.h
+++ b/arch/powerpc/include/asm/processor.h
@@ -237,6 +237,8 @@ struct thread_struct {
unsigned long evr[32]; /* upper 32-bits of SPE regs */
u64 acc; /* Accumulator */
unsigned long spefscr; /* SPE & eFP status */
+ unsigned long spefscr_last; /* SPEFSCR value on last prctl
+ call or trap return */
int used_spe; /* set if process has used spe */
#endif /* CONFIG_SPE */
#ifdef CONFIG_PPC_TRANSACTIONAL_MEM
@@ -303,7 +305,9 @@ struct thread_struct {
(_ALIGN_UP(sizeof(init_thread_info), 16) + (unsigned long) &init_stack)
#ifdef CONFIG_SPE
-#define SPEFSCR_INIT .spefscr = SPEFSCR_FINVE | SPEFSCR_FDBZE | SPEFSCR_FUNFE | SPEFSCR_FOVFE,
+#define SPEFSCR_INIT \
+ .spefscr = SPEFSCR_FINVE | SPEFSCR_FDBZE | SPEFSCR_FUNFE | SPEFSCR_FOVFE, \
+ .spefscr_last = SPEFSCR_FINVE | SPEFSCR_FDBZE | SPEFSCR_FUNFE | SPEFSCR_FOVFE,
#else
#define SPEFSCR_INIT
#endif
diff --git a/arch/powerpc/kernel/process.c b/arch/powerpc/kernel/process.c
index 96d2fdf..e3b91f1 100644
--- a/arch/powerpc/kernel/process.c
+++ b/arch/powerpc/kernel/process.c
@@ -1151,6 +1151,7 @@ int set_fpexc_mode(struct task_struct *tsk, unsigned int val)
if (val & PR_FP_EXC_SW_ENABLE) {
#ifdef CONFIG_SPE
if (cpu_has_feature(CPU_FTR_SPE)) {
+ tsk->thread.spefscr_last = mfspr(SPRN_SPEFSCR);
tsk->thread.fpexc_mode = val &
(PR_FP_EXC_SW_ENABLE | PR_FP_ALL_EXCEPT);
return 0;
@@ -1182,9 +1183,10 @@ int get_fpexc_mode(struct task_struct *tsk, unsigned long adr)
if (tsk->thread.fpexc_mode & PR_FP_EXC_SW_ENABLE)
#ifdef CONFIG_SPE
- if (cpu_has_feature(CPU_FTR_SPE))
+ if (cpu_has_feature(CPU_FTR_SPE)) {
+ tsk->thread.spefscr_last = mfspr(SPRN_SPEFSCR);
val = tsk->thread.fpexc_mode;
- else
+ } else
return -EINVAL;
#else
return -EINVAL;
diff --git a/arch/powerpc/math-emu/math_efp.c b/arch/powerpc/math-emu/math_efp.c
index a73f088..59835c6 100644
--- a/arch/powerpc/math-emu/math_efp.c
+++ b/arch/powerpc/math-emu/math_efp.c
@@ -630,9 +630,27 @@ update_ccr:
regs->ccr |= (IR << ((7 - ((speinsn >> 23) & 0x7)) << 2));
update_regs:
- __FPU_FPSCR &= ~FP_EX_MASK;
+ /*
+ * If the "invalid" exception sticky bit was set by the
+ * processor for non-finite input, but was not set before the
+ * instruction being emulated, clear it. Likewise for the
+ * "underflow" bit, which may have been set by the processor
+ * for exact underflow, not just inexact underflow when the
+ * flag should be set for IEEE 754 semantics. Other sticky
+ * exceptions will only be set by the processor when they are
+ * correct according to IEEE 754 semantics, and we must not
+ * clear sticky bits that were already set before the emulated
+ * instruction as they represent the user-visible sticky
+ * exception status. "inexact" traps to kernel are not
+ * required for IEEE semantics and are not enabled by default,
+ * so the "inexact" sticky bit may have been set by a previous
+ * instruction without the kernel being aware of it.
+ */
+ __FPU_FPSCR
+ &= ~(FP_EX_INVALID | FP_EX_UNDERFLOW) | current->thread.spefscr_last;
__FPU_FPSCR |= (FP_CUR_EXCEPTIONS & FP_EX_MASK);
mtspr(SPRN_SPEFSCR, __FPU_FPSCR);
+ current->thread.spefscr_last = __FPU_FPSCR;
current->thread.evr[fc] = vc.wp[0];
regs->gpr[fc] = vc.wp[1];
--
Joseph S. Myers
joseph@codesourcery.com
^ permalink raw reply related
* [PATCH 2/6] powerpc: fix e500 SPE float rounding inexactness detection
From: Joseph S. Myers @ 2013-11-04 16:53 UTC (permalink / raw)
To: linuxppc-dev; +Cc: Liu Yu, linux-kernel, Shan Hai
In-Reply-To: <Pine.LNX.4.64.1311041649250.4290@digraph.polyomino.org.uk>
From: Joseph Myers <joseph@codesourcery.com>
The e500 SPE floating-point emulation code for the rounding modes
rounding to positive or negative infinity (which may not be
implemented in hardware) tries to avoid emulating rounding if the
result was inexact. However, it tests inexactness using the sticky
bit with the cumulative result of previous operations, rather than
with the non-sticky bits relating to the operation that generated the
interrupt. Furthermore, when a vector operation generates the
interrupt, it's possible that only one of the low and high parts is
inexact, and so only that part should have rounding emulated. This
results in incorrect rounding of exact results in these modes when the
sticky bit is set from a previous operation.
(I'm not sure why the rounding interrupts are generated at all when
the result is exact, but empirically the hardware does generate them.)
This patch checks for inexactness using the correct bits of SPEFSCR,
and ensures that rounding only occurs when the relevant part of the
result was actually inexact.
Signed-off-by: Joseph Myers <joseph@codesourcery.com>
---
Previous submission: <http://lkml.org/lkml/2013/10/4/497>.
diff --git a/arch/powerpc/math-emu/math_efp.c b/arch/powerpc/math-emu/math_efp.c
index 59835c6..ecdf35d 100644
--- a/arch/powerpc/math-emu/math_efp.c
+++ b/arch/powerpc/math-emu/math_efp.c
@@ -680,7 +680,8 @@ int speround_handler(struct pt_regs *regs)
{
union dw_union fgpr;
int s_lo, s_hi;
- unsigned long speinsn, type, fc;
+ int lo_inexact, hi_inexact;
+ unsigned long speinsn, type, fc, fptype;
if (get_user(speinsn, (unsigned int __user *) regs->nip))
return -EFAULT;
@@ -693,8 +694,12 @@ int speround_handler(struct pt_regs *regs)
__FPU_FPSCR = mfspr(SPRN_SPEFSCR);
pr_debug("speinsn:%08lx spefscr:%08lx\n", speinsn, __FPU_FPSCR);
+ fptype = (speinsn >> 5) & 0x7;
+
/* No need to round if the result is exact */
- if (!(__FPU_FPSCR & FP_EX_INEXACT))
+ lo_inexact = __FPU_FPSCR & (SPEFSCR_FG | SPEFSCR_FX);
+ hi_inexact = __FPU_FPSCR & (SPEFSCR_FGH | SPEFSCR_FXH);
+ if (!(lo_inexact || (hi_inexact && fptype == VCT)))
return 0;
fc = (speinsn >> 21) & 0x1f;
@@ -705,7 +710,7 @@ int speround_handler(struct pt_regs *regs)
pr_debug("round fgpr: %08x %08x\n", fgpr.wp[0], fgpr.wp[1]);
- switch ((speinsn >> 5) & 0x7) {
+ switch (fptype) {
/* Since SPE instructions on E500 core can handle round to nearest
* and round toward zero with IEEE-754 complied, we just need
* to handle round toward +Inf and round toward -Inf by software.
@@ -728,11 +733,15 @@ int speround_handler(struct pt_regs *regs)
case VCT:
if (FP_ROUNDMODE == FP_RND_PINF) {
- if (!s_lo) fgpr.wp[1]++; /* Z_low > 0, choose Z1 */
- if (!s_hi) fgpr.wp[0]++; /* Z_high word > 0, choose Z1 */
+ if (lo_inexact && !s_lo)
+ fgpr.wp[1]++; /* Z_low > 0, choose Z1 */
+ if (hi_inexact && !s_hi)
+ fgpr.wp[0]++; /* Z_high word > 0, choose Z1 */
} else { /* round to -Inf */
- if (s_lo) fgpr.wp[1]++; /* Z_low < 0, choose Z2 */
- if (s_hi) fgpr.wp[0]++; /* Z_high < 0, choose Z2 */
+ if (lo_inexact && s_lo)
+ fgpr.wp[1]++; /* Z_low < 0, choose Z2 */
+ if (hi_inexact && s_hi)
+ fgpr.wp[0]++; /* Z_high < 0, choose Z2 */
}
break;
--
Joseph S. Myers
joseph@codesourcery.com
^ permalink raw reply related
* [PATCH 3/6] math-emu: fix floating-point to integer unsigned saturation
From: Joseph S. Myers @ 2013-11-04 16:53 UTC (permalink / raw)
To: linuxppc-dev; +Cc: Liu Yu, linux-kernel, Shan Hai
In-Reply-To: <Pine.LNX.4.64.1311041649250.4290@digraph.polyomino.org.uk>
From: Joseph Myers <joseph@codesourcery.com>
The math-emu macros _FP_TO_INT and _FP_TO_INT_ROUND are supposed to
saturate their results for out-of-range arguments, except in the case
rsigned == 2 (when instead the low bits of the result are taken).
However, in the case rsigned == 0 (converting to unsigned integers),
they mistakenly produce 0 for positive results and the maximum
unsigned integer for negative results, the opposite of correct
unsigned saturation. This patch fixes the logic.
Signed-off-by: Joseph Myers <joseph@codesourcery.com>
---
Previous submission: <http://lkml.org/lkml/2013/10/8/694>.
I have made the corresponding changes to the glibc/libgcc copy of this
code, given that it would be desirable to resync the Linux and
glibc/libgcc copies (the latter has had many enhancements and bug
fixes since it was copied into Linux), although strictly this
incorrect saturation is only a bug when trying to emulate particular
instruction semantics, not when used in userspace to implement C
operations where the results of out-of-range conversions are
unspecified or undefined.
diff --git a/include/math-emu/op-common.h b/include/math-emu/op-common.h
index 9696a5e..70fe5e9 100644
--- a/include/math-emu/op-common.h
+++ b/include/math-emu/op-common.h
@@ -685,7 +685,7 @@ do { \
else \
{ \
r = 0; \
- if (X##_s) \
+ if (!X##_s) \
r = ~r; \
} \
FP_SET_EXCEPTION(FP_EX_INVALID); \
@@ -762,7 +762,7 @@ do { \
if (!rsigned) \
{ \
r = 0; \
- if (X##_s) \
+ if (!X##_s) \
r = ~r; \
} \
else if (rsigned != 2) \
--
Joseph S. Myers
joseph@codesourcery.com
^ permalink raw reply related
* [PATCH 4/6] math-emu: fix floating-point to integer overflow detection
From: Joseph S. Myers @ 2013-11-04 16:54 UTC (permalink / raw)
To: linuxppc-dev; +Cc: Liu Yu, linux-kernel, Shan Hai
In-Reply-To: <Pine.LNX.4.64.1311041649250.4290@digraph.polyomino.org.uk>
From: Joseph Myers <joseph@codesourcery.com>
On overflow, the math-emu macro _FP_TO_INT_ROUND tries to saturate its
result (subject to the value of rsigned specifying the desired
overflow semantics). However, if the rounding step has the effect of
increasing the exponent so as to cause overflow (if the rounded result
is 1 larger than the largest positive value with the given number of
bits, allowing for signedness), the overflow does not get detected,
meaning that for unsigned results 0 is produced instead of the maximum
unsigned integer with the give number of bits, without an exception
being raised for overflow, and that for signed results the minimum
(negative) value is produced instead of the maximum (positive) value,
again without an exception. This patch makes the code check for
rounding increasing the exponent and adjusts the exponent value as
needed for the overflow check.
Signed-off-by: Joseph Myers <joseph@codesourcery.com>
---
Previous submission: <http://lkml.org/lkml/2013/10/8/700>.
This macro is not present in the glibc/libgcc version of the code. It
remains the case both before and after this patch that the conversions
wrongly treat a signed result of the most negative integer as an
overflow, when actually only that integer minus 1 or smaller should be
an overflow, although this only means an incorrect exception rather
than affecting the value returned; that was one of the bugs I fixed in
the glibc/libgcc version of this code in 2006 (as part of a major
overhaul of the code including various interface changes, so not
trivially backportable to the kernel version).
diff --git a/include/math-emu/op-common.h b/include/math-emu/op-common.h
index 70fe5e9..6bdf8c6 100644
--- a/include/math-emu/op-common.h
+++ b/include/math-emu/op-common.h
@@ -743,12 +743,17 @@ do { \
} \
else \
{ \
+ int _lz0, _lz1; \
if (X##_e <= -_FP_WORKBITS - 1) \
_FP_FRAC_SET_##wc(X, _FP_MINFRAC_##wc); \
else \
_FP_FRAC_SRS_##wc(X, _FP_FRACBITS_##fs - 1 - X##_e, \
_FP_WFRACBITS_##fs); \
+ _FP_FRAC_CLZ_##wc(_lz0, X); \
_FP_ROUND(wc, X); \
+ _FP_FRAC_CLZ_##wc(_lz1, X); \
+ if (_lz1 < _lz0) \
+ X##_e++; /* For overflow detection. */ \
_FP_FRAC_SRL_##wc(X, _FP_WORKBITS); \
_FP_FRAC_ASSEMBLE_##wc(r, X, rsize); \
} \
--
Joseph S. Myers
joseph@codesourcery.com
^ permalink raw reply related
* [PATCH 5/6] powerpc: fix e500 SPE float to integer and fixed-point conversions
From: Joseph S. Myers @ 2013-11-04 16:54 UTC (permalink / raw)
To: linuxppc-dev; +Cc: Liu Yu, linux-kernel, Shan Hai
In-Reply-To: <Pine.LNX.4.64.1311041649250.4290@digraph.polyomino.org.uk>
From: Joseph Myers <joseph@codesourcery.com>
The e500 SPE floating-point emulation code has several problems in how
it handles conversions to integer and fixed-point fractional types.
There are the following 20 relevant instructions. These can convert
to signed or unsigned 32-bit integers, either rounding towards zero
(as correct for C casts from floating-point to integer) or according
to the current rounding mode, or to signed or unsigned 32-bit
fixed-point values (values in the range [-1, 1) or [0, 1)). For
conversion from double precision there are also instructions to
convert to 64-bit integers, rounding towards zero, although as far as
I know those instructions are completely theoretical (they are only
defined for implementations that support both SPE and classic 64-bit,
and I'm not aware of any such hardware even though the architecture
definition permits that combination).
#define EFSCTUI 0x2d4
#define EFSCTSI 0x2d5
#define EFSCTUF 0x2d6
#define EFSCTSF 0x2d7
#define EFSCTUIZ 0x2d8
#define EFSCTSIZ 0x2da
#define EVFSCTUI 0x294
#define EVFSCTSI 0x295
#define EVFSCTUF 0x296
#define EVFSCTSF 0x297
#define EVFSCTUIZ 0x298
#define EVFSCTSIZ 0x29a
#define EFDCTUIDZ 0x2ea
#define EFDCTSIDZ 0x2eb
#define EFDCTUI 0x2f4
#define EFDCTSI 0x2f5
#define EFDCTUF 0x2f6
#define EFDCTSF 0x2f7
#define EFDCTUIZ 0x2f8
#define EFDCTSIZ 0x2fa
The emulation code, for the instructions that come in variants
rounding either towards zero or according to the current rounding
direction, uses "if (func & 0x4)" as a condition for using _FP_ROUND
(otherwise _FP_ROUND_ZERO is used). The condition is correct, but the
code it controls isn't. Whether _FP_ROUND or _FP_ROUND_ZERO is used
makes no difference, as the effect of those soft-fp macros is to round
an intermediate floating-point result using the low three bits (the
last one sticky) of the working format. As these operations are
dealing with a freshly unpacked floating-point input, those low bits
are zero and no rounding occurs. The emulation code then uses the
FP_TO_INT_* macros for the actual integer conversion, with the effect
of always rounding towards zero; for rounding according to the current
rounding direction, it should be using FP_TO_INT_ROUND_*.
The instructions in question have semantics defined (in the Power ISA
documents) for out-of-range values and NaNs: out-of-range values
saturate and NaNs are converted to zero. The emulation does nothing
to follow those semantics for NaNs (the soft-fp handling is to treat
them as infinities), and messes up the saturation semantics. For
single-precision conversion to integers, (((func & 0x3) != 0) || SB_s)
is the condition used for doing a signed conversion. The first part
is correct, but the second isn't: negative numbers should result in
saturation to 0 when converted to unsigned. Double-precision
conversion to 64-bit integers correctly uses ((func & 0x1) == 0).
Double-precision conversion to 32-bit integers uses (((func & 0x3) !=
0) || DB_s), with correct first part and incorrect second part. And
vector float conversion to integers uses (((func & 0x3) != 0) ||
SB0_s) (and similar for the other vector element), where the sign bit
check is again wrong.
The incorrect handling of negative numbers converted to unsigned was
introduced in commit afc0a07d4a283599ac3a6a31d7454e9baaeccca0. The
rationale given there was a C testcase with cast from float to
unsigned int. Conversion of out-of-range floating-point numbers to
integer types in C is undefined behavior in the base standard, defined
in Annex F to produce an unspecified value. That is, the C testcase
used to justify that patch is incorrect - there is no ISO C
requirement for a particular value resulting from this conversion -
and in any case, the correct semantics for such emulation are the
semantics for the instruction (unsigned saturation, which is what it
does in hardware when the emulation is disabled).
The conversion to fixed-point values has its own problems. That code
doesn't try to do a full emulation; it relies on the trap handler only
being called for arguments that are infinities, NaNs, subnormal or out
of range. That's fine, but the logic ((vb.wp[1] >> 23) == 0xff &&
((vb.wp[1] & 0x7fffff) > 0)) for NaN detection won't detect negative
NaNs as being NaNs (the same applies for the double-precision case),
and subnormals are mapped to 0 rather than respecting the rounding
mode; the code should also explicitly raise the "invalid" exception.
The code for vectors works by executing the scalar float instruction
with the trapping disabled, meaning at least subnormals won't be
handled correctly.
As well as all those problems in the main emulation code, the rounding
handler - used to emulate rounding upward and downward when not
supported in hardware and when no higher priority exception occurred -
has its own problems.
* It gets called in some cases even for the instructions rounding to
zero, and then acts according to the current rounding mode when it
should just leave alone the truncated result provided by hardware.
* It presumes that the result is a single-precision, double-precision
or single-precision vector as appropriate for the instruction type,
determines the sign of the result accordingly, and then adjusts the
result based on that sign and the rounding mode.
- In the single-precision cases at least the sign determination for
an integer result is the same as for a floating-point result; in
the double-precision case, converted to 32-bit integer or fixed
point, the sign of a double-precision value is in the high part of
the register but it's the low part of the register that has the
result of the conversion.
- If the result is unsigned fixed-point, its sign may be wrongly
determined as negative (does not actually cause problems, because
inexact unsigned fixed-point results with the high bit set can
only appear when converting from double, in which case the sign
determination is instead wrongly using the high part of the
register).
- If the sign of the result is correctly determined as negative, any
adjustment required to change the truncated result to one correct
for the rounding mode should be in the opposite direction for
two's-complement integers as for sign-magnitude floating-point
values.
- And if the integer result is zero, the correct sign can only be
determined by examining the original operand, and not at all (as
far as I can tell) if the operand and result are the same
register.
This patch fixes all these problems (as far as possible, given the
inability to determine the correct sign in the rounding handler when
the truncated result is 0, the conversion is to a signed type and the
truncated result has overwritten the original operand). Conversion to
fixed-point now uses full emulation, and does not use "asm" in the
vector case; the semantics are exactly those of converting to integer
according to the current rounding direction, once the exponent has
been adjusted, so the code makes such an adjustment then uses the
FP_TO_INT_ROUND macros.
The testcase I used for verifying that the instructions (other than
the theoretical conversions to 64-bit integers) produce the correct
results is at <http://lkml.org/lkml/2013/10/8/708>.
Signed-off-by: Joseph Myers <joseph@codesourcery.com>
---
Previous submission: <http://lkml.org/lkml/2013/10/8/705>.
diff --git a/arch/powerpc/math-emu/math_efp.c b/arch/powerpc/math-emu/math_efp.c
index ecdf35d..01a0abb 100644
--- a/arch/powerpc/math-emu/math_efp.c
+++ b/arch/powerpc/math-emu/math_efp.c
@@ -275,21 +275,13 @@ int do_spe_mathemu(struct pt_regs *regs)
case EFSCTSF:
case EFSCTUF:
- if (!((vb.wp[1] >> 23) == 0xff && ((vb.wp[1] & 0x7fffff) > 0))) {
- /* NaN */
- if (((vb.wp[1] >> 23) & 0xff) == 0) {
- /* denorm */
- vc.wp[1] = 0x0;
- } else if ((vb.wp[1] >> 31) == 0) {
- /* positive normal */
- vc.wp[1] = (func == EFSCTSF) ?
- 0x7fffffff : 0xffffffff;
- } else { /* negative normal */
- vc.wp[1] = (func == EFSCTSF) ?
- 0x80000000 : 0x0;
- }
- } else { /* rB is NaN */
- vc.wp[1] = 0x0;
+ if (SB_c == FP_CLS_NAN) {
+ vc.wp[1] = 0;
+ FP_SET_EXCEPTION(FP_EX_INVALID);
+ } else {
+ SB_e += (func == EFSCTSF ? 31 : 32);
+ FP_TO_INT_ROUND_S(vc.wp[1], SB, 32,
+ (func == EFSCTSF));
}
goto update_regs;
@@ -306,16 +298,25 @@ int do_spe_mathemu(struct pt_regs *regs)
}
case EFSCTSI:
- case EFSCTSIZ:
case EFSCTUI:
+ if (SB_c == FP_CLS_NAN) {
+ vc.wp[1] = 0;
+ FP_SET_EXCEPTION(FP_EX_INVALID);
+ } else {
+ FP_TO_INT_ROUND_S(vc.wp[1], SB, 32,
+ ((func & 0x3) != 0));
+ }
+ goto update_regs;
+
+ case EFSCTSIZ:
case EFSCTUIZ:
- if (func & 0x4) {
- _FP_ROUND(1, SB);
+ if (SB_c == FP_CLS_NAN) {
+ vc.wp[1] = 0;
+ FP_SET_EXCEPTION(FP_EX_INVALID);
} else {
- _FP_ROUND_ZERO(1, SB);
+ FP_TO_INT_S(vc.wp[1], SB, 32,
+ ((func & 0x3) != 0));
}
- FP_TO_INT_S(vc.wp[1], SB, 32,
- (((func & 0x3) != 0) || SB_s));
goto update_regs;
default:
@@ -404,22 +405,13 @@ cmp_s:
case EFDCTSF:
case EFDCTUF:
- if (!((vb.wp[0] >> 20) == 0x7ff &&
- ((vb.wp[0] & 0xfffff) > 0 || (vb.wp[1] > 0)))) {
- /* not a NaN */
- if (((vb.wp[0] >> 20) & 0x7ff) == 0) {
- /* denorm */
- vc.wp[1] = 0x0;
- } else if ((vb.wp[0] >> 31) == 0) {
- /* positive normal */
- vc.wp[1] = (func == EFDCTSF) ?
- 0x7fffffff : 0xffffffff;
- } else { /* negative normal */
- vc.wp[1] = (func == EFDCTSF) ?
- 0x80000000 : 0x0;
- }
- } else { /* NaN */
- vc.wp[1] = 0x0;
+ if (DB_c == FP_CLS_NAN) {
+ vc.wp[1] = 0;
+ FP_SET_EXCEPTION(FP_EX_INVALID);
+ } else {
+ DB_e += (func == EFDCTSF ? 31 : 32);
+ FP_TO_INT_ROUND_D(vc.wp[1], DB, 32,
+ (func == EFDCTSF));
}
goto update_regs;
@@ -437,21 +429,35 @@ cmp_s:
case EFDCTUIDZ:
case EFDCTSIDZ:
- _FP_ROUND_ZERO(2, DB);
- FP_TO_INT_D(vc.dp[0], DB, 64, ((func & 0x1) == 0));
+ if (DB_c == FP_CLS_NAN) {
+ vc.dp[0] = 0;
+ FP_SET_EXCEPTION(FP_EX_INVALID);
+ } else {
+ FP_TO_INT_D(vc.dp[0], DB, 64,
+ ((func & 0x1) == 0));
+ }
goto update_regs;
case EFDCTUI:
case EFDCTSI:
+ if (DB_c == FP_CLS_NAN) {
+ vc.wp[1] = 0;
+ FP_SET_EXCEPTION(FP_EX_INVALID);
+ } else {
+ FP_TO_INT_ROUND_D(vc.wp[1], DB, 32,
+ ((func & 0x3) != 0));
+ }
+ goto update_regs;
+
case EFDCTUIZ:
case EFDCTSIZ:
- if (func & 0x4) {
- _FP_ROUND(2, DB);
+ if (DB_c == FP_CLS_NAN) {
+ vc.wp[1] = 0;
+ FP_SET_EXCEPTION(FP_EX_INVALID);
} else {
- _FP_ROUND_ZERO(2, DB);
+ FP_TO_INT_D(vc.wp[1], DB, 32,
+ ((func & 0x3) != 0));
}
- FP_TO_INT_D(vc.wp[1], DB, 32,
- (((func & 0x3) != 0) || DB_s));
goto update_regs;
default:
@@ -556,37 +562,60 @@ cmp_d:
cmp = -1;
goto cmp_vs;
- case EVFSCTSF:
- __asm__ __volatile__ ("mtspr 512, %4\n"
- "efsctsf %0, %2\n"
- "efsctsf %1, %3\n"
- : "=r" (vc.wp[0]), "=r" (vc.wp[1])
- : "r" (vb.wp[0]), "r" (vb.wp[1]), "r" (0));
- goto update_regs;
-
case EVFSCTUF:
- __asm__ __volatile__ ("mtspr 512, %4\n"
- "efsctuf %0, %2\n"
- "efsctuf %1, %3\n"
- : "=r" (vc.wp[0]), "=r" (vc.wp[1])
- : "r" (vb.wp[0]), "r" (vb.wp[1]), "r" (0));
+ case EVFSCTSF:
+ if (SB0_c == FP_CLS_NAN) {
+ vc.wp[0] = 0;
+ FP_SET_EXCEPTION(FP_EX_INVALID);
+ } else {
+ SB0_e += (func == EVFSCTSF ? 31 : 32);
+ FP_TO_INT_ROUND_S(vc.wp[0], SB0, 32,
+ (func == EVFSCTSF));
+ }
+ if (SB1_c == FP_CLS_NAN) {
+ vc.wp[1] = 0;
+ FP_SET_EXCEPTION(FP_EX_INVALID);
+ } else {
+ SB1_e += (func == EVFSCTSF ? 31 : 32);
+ FP_TO_INT_ROUND_S(vc.wp[1], SB1, 32,
+ (func == EVFSCTSF));
+ }
goto update_regs;
case EVFSCTUI:
case EVFSCTSI:
+ if (SB0_c == FP_CLS_NAN) {
+ vc.wp[0] = 0;
+ FP_SET_EXCEPTION(FP_EX_INVALID);
+ } else {
+ FP_TO_INT_ROUND_S(vc.wp[0], SB0, 32,
+ ((func & 0x3) != 0));
+ }
+ if (SB1_c == FP_CLS_NAN) {
+ vc.wp[1] = 0;
+ FP_SET_EXCEPTION(FP_EX_INVALID);
+ } else {
+ FP_TO_INT_ROUND_S(vc.wp[1], SB1, 32,
+ ((func & 0x3) != 0));
+ }
+ goto update_regs;
+
case EVFSCTUIZ:
case EVFSCTSIZ:
- if (func & 0x4) {
- _FP_ROUND(1, SB0);
- _FP_ROUND(1, SB1);
+ if (SB0_c == FP_CLS_NAN) {
+ vc.wp[0] = 0;
+ FP_SET_EXCEPTION(FP_EX_INVALID);
} else {
- _FP_ROUND_ZERO(1, SB0);
- _FP_ROUND_ZERO(1, SB1);
+ FP_TO_INT_S(vc.wp[0], SB0, 32,
+ ((func & 0x3) != 0));
+ }
+ if (SB1_c == FP_CLS_NAN) {
+ vc.wp[1] = 0;
+ FP_SET_EXCEPTION(FP_EX_INVALID);
+ } else {
+ FP_TO_INT_S(vc.wp[1], SB1, 32,
+ ((func & 0x3) != 0));
}
- FP_TO_INT_S(vc.wp[0], SB0, 32,
- (((func & 0x3) != 0) || SB0_s));
- FP_TO_INT_S(vc.wp[1], SB1, 32,
- (((func & 0x3) != 0) || SB1_s));
goto update_regs;
default:
@@ -681,14 +710,16 @@ int speround_handler(struct pt_regs *regs)
union dw_union fgpr;
int s_lo, s_hi;
int lo_inexact, hi_inexact;
- unsigned long speinsn, type, fc, fptype;
+ int fp_result;
+ unsigned long speinsn, type, fb, fc, fptype, func;
if (get_user(speinsn, (unsigned int __user *) regs->nip))
return -EFAULT;
if ((speinsn >> 26) != 4)
return -EINVAL; /* not an spe instruction */
- type = insn_type(speinsn & 0x7ff);
+ func = speinsn & 0x7ff;
+ type = insn_type(func);
if (type == XCR) return -ENOSYS;
__FPU_FPSCR = mfspr(SPRN_SPEFSCR);
@@ -708,6 +739,65 @@ int speround_handler(struct pt_regs *regs)
fgpr.wp[0] = current->thread.evr[fc];
fgpr.wp[1] = regs->gpr[fc];
+ fb = (speinsn >> 11) & 0x1f;
+ switch (func) {
+ case EFSCTUIZ:
+ case EFSCTSIZ:
+ case EVFSCTUIZ:
+ case EVFSCTSIZ:
+ case EFDCTUIDZ:
+ case EFDCTSIDZ:
+ case EFDCTUIZ:
+ case EFDCTSIZ:
+ /*
+ * These instructions always round to zero,
+ * independent of the rounding mode.
+ */
+ return 0;
+
+ case EFSCTUI:
+ case EFSCTUF:
+ case EVFSCTUI:
+ case EVFSCTUF:
+ case EFDCTUI:
+ case EFDCTUF:
+ fp_result = 0;
+ s_lo = 0;
+ s_hi = 0;
+ break;
+
+ case EFSCTSI:
+ case EFSCTSF:
+ fp_result = 0;
+ /* Recover the sign of a zero result if possible. */
+ if (fgpr.wp[1] == 0)
+ s_lo = regs->gpr[fb] & SIGN_BIT_S;
+ break;
+
+ case EVFSCTSI:
+ case EVFSCTSF:
+ fp_result = 0;
+ /* Recover the sign of a zero result if possible. */
+ if (fgpr.wp[1] == 0)
+ s_lo = regs->gpr[fb] & SIGN_BIT_S;
+ if (fgpr.wp[0] == 0)
+ s_hi = current->thread.evr[fb] & SIGN_BIT_S;
+ break;
+
+ case EFDCTSI:
+ case EFDCTSF:
+ fp_result = 0;
+ s_hi = s_lo;
+ /* Recover the sign of a zero result if possible. */
+ if (fgpr.wp[1] == 0)
+ s_hi = current->thread.evr[fb] & SIGN_BIT_S;
+ break;
+
+ default:
+ fp_result = 1;
+ break;
+ }
+
pr_debug("round fgpr: %08x %08x\n", fgpr.wp[0], fgpr.wp[1]);
switch (fptype) {
@@ -719,15 +809,30 @@ int speround_handler(struct pt_regs *regs)
if ((FP_ROUNDMODE) == FP_RND_PINF) {
if (!s_lo) fgpr.wp[1]++; /* Z > 0, choose Z1 */
} else { /* round to -Inf */
- if (s_lo) fgpr.wp[1]++; /* Z < 0, choose Z2 */
+ if (s_lo) {
+ if (fp_result)
+ fgpr.wp[1]++; /* Z < 0, choose Z2 */
+ else
+ fgpr.wp[1]--; /* Z < 0, choose Z2 */
+ }
}
break;
case DPFP:
if (FP_ROUNDMODE == FP_RND_PINF) {
- if (!s_hi) fgpr.dp[0]++; /* Z > 0, choose Z1 */
+ if (!s_hi) {
+ if (fp_result)
+ fgpr.dp[0]++; /* Z > 0, choose Z1 */
+ else
+ fgpr.wp[1]++; /* Z > 0, choose Z1 */
+ }
} else { /* round to -Inf */
- if (s_hi) fgpr.dp[0]++; /* Z < 0, choose Z2 */
+ if (s_hi) {
+ if (fp_result)
+ fgpr.dp[0]++; /* Z < 0, choose Z2 */
+ else
+ fgpr.wp[1]--; /* Z < 0, choose Z2 */
+ }
}
break;
@@ -738,10 +843,18 @@ int speround_handler(struct pt_regs *regs)
if (hi_inexact && !s_hi)
fgpr.wp[0]++; /* Z_high word > 0, choose Z1 */
} else { /* round to -Inf */
- if (lo_inexact && s_lo)
- fgpr.wp[1]++; /* Z_low < 0, choose Z2 */
- if (hi_inexact && s_hi)
- fgpr.wp[0]++; /* Z_high < 0, choose Z2 */
+ if (lo_inexact && s_lo) {
+ if (fp_result)
+ fgpr.wp[1]++; /* Z_low < 0, choose Z2 */
+ else
+ fgpr.wp[1]--; /* Z_low < 0, choose Z2 */
+ }
+ if (hi_inexact && s_hi) {
+ if (fp_result)
+ fgpr.wp[0]++; /* Z_high < 0, choose Z2 */
+ else
+ fgpr.wp[0]--; /* Z_high < 0, choose Z2 */
+ }
}
break;
--
Joseph S. Myers
joseph@codesourcery.com
^ permalink raw reply related
* [PATCH 6/6] powerpc: fix e500 SPE float SIGFPE generation
From: Joseph S. Myers @ 2013-11-04 16:55 UTC (permalink / raw)
To: linuxppc-dev; +Cc: Liu Yu, linux-kernel, Shan Hai
In-Reply-To: <Pine.LNX.4.64.1311041649250.4290@digraph.polyomino.org.uk>
From: Joseph Myers <joseph@codesourcery.com>
The e500 SPE floating-point emulation code is called from
SPEFloatingPointException and SPEFloatingPointRoundException in
arch/powerpc/kernel/traps.c. Those functions have support for
generating SIGFPE, but do_spe_mathemu and speround_handler don't
generate a return value to indicate that this should be done. Such a
return value should depend on whether an exception is raised that has
been set via prctl to generate SIGFPE. This patch adds the relevant
logic in these functions so that SIGFPE is generated as expected by
the glibc testsuite.
Signed-off-by: Joseph Myers <joseph@codesourcery.com>
---
Previous submission: <http://lkml.org/lkml/2013/10/10/626>.
diff --git a/arch/powerpc/math-emu/math_efp.c b/arch/powerpc/math-emu/math_efp.c
index 01a0abb..28337c9 100644
--- a/arch/powerpc/math-emu/math_efp.c
+++ b/arch/powerpc/math-emu/math_efp.c
@@ -20,6 +20,7 @@
*/
#include <linux/types.h>
+#include <linux/prctl.h>
#include <asm/uaccess.h>
#include <asm/reg.h>
@@ -691,6 +692,23 @@ update_regs:
pr_debug("va: %08x %08x\n", va.wp[0], va.wp[1]);
pr_debug("vb: %08x %08x\n", vb.wp[0], vb.wp[1]);
+ if (current->thread.fpexc_mode & PR_FP_EXC_SW_ENABLE) {
+ if ((FP_CUR_EXCEPTIONS & FP_EX_DIVZERO)
+ && (current->thread.fpexc_mode & PR_FP_EXC_DIV))
+ return 1;
+ if ((FP_CUR_EXCEPTIONS & FP_EX_OVERFLOW)
+ && (current->thread.fpexc_mode & PR_FP_EXC_OVF))
+ return 1;
+ if ((FP_CUR_EXCEPTIONS & FP_EX_UNDERFLOW)
+ && (current->thread.fpexc_mode & PR_FP_EXC_UND))
+ return 1;
+ if ((FP_CUR_EXCEPTIONS & FP_EX_INEXACT)
+ && (current->thread.fpexc_mode & PR_FP_EXC_RES))
+ return 1;
+ if ((FP_CUR_EXCEPTIONS & FP_EX_INVALID)
+ && (current->thread.fpexc_mode & PR_FP_EXC_INV))
+ return 1;
+ }
return 0;
illegal:
@@ -867,6 +885,8 @@ int speround_handler(struct pt_regs *regs)
pr_debug(" to fgpr: %08x %08x\n", fgpr.wp[0], fgpr.wp[1]);
+ if (current->thread.fpexc_mode & PR_FP_EXC_SW_ENABLE)
+ return (current->thread.fpexc_mode & PR_FP_EXC_RES) ? 1 : 0;
return 0;
}
--
Joseph S. Myers
joseph@codesourcery.com
^ permalink raw reply related
* Re: [RFC] arch: Introduce new TSO memory barrier smp_tmb()
From: Paul E. McKenney @ 2013-11-04 16:34 UTC (permalink / raw)
To: Will Deacon
Cc: Michael Neuling, Mathieu Desnoyers, Peter Zijlstra, Oleg Nesterov,
LKML, Linux PPC dev, Anton Blanchard, Frederic Weisbecker,
Victor Kaplansky, Linus Torvalds
In-Reply-To: <20131104110553.GA8595@mudshark.cambridge.arm.com>
On Mon, Nov 04, 2013 at 11:05:53AM +0000, Will Deacon wrote:
> On Sun, Nov 03, 2013 at 11:34:00PM +0000, Linus Torvalds wrote:
> > So it would *kind* of act like a "smp_wmb() + smp_rmb()", but the
> > problem is that a "smp_rmb()" doesn't really "attach" to the preceding
> > write.
>
> Agreed.
>
> > This is analogous to a "acquire" operation: you cannot make an
> > "acquire" barrier, because it's not a barrier *between* two ops, it's
> > associated with one particular op.
> >
> > So what I *think* you actually really really want is a "store with
> > release consistency, followed by a write barrier".
>
> How does that order reads against reads? (Paul mentioned this as a
> requirement). I not clear about the use case for this, so perhaps there is a
> dependency that I'm not aware of.
An smp_store_with_release_semantics() orders against prior reads -and-
writes. It maps to barrier() for x86, stlr for ARM, and lwsync for
PowerPC, as called out in my prototype definitions.
> > In TSO, afaik all stores have release consistency, and all writes are
> > ordered, which is why this is a no-op in TSO. And x86 also has that
> > "all stores have release consistency, and all writes are ordered"
> > model, even if TSO doesn't really describe the x86 model.
> >
> > But on ARM64, for example, I think you'd really want the store itself
> > to be done with "stlr" (store with release), and then follow up with a
> > "dsb st" after that.
>
> So a dsb is pretty heavyweight here (it prevents execution of *any* further
> instructions until all preceeding stores have completed, as well as
> ensuring completion of any ongoing cache flushes). In conjunction with the
> store-release, that's going to hold everything up until the store-release
> (and therefore any preceeding memory accesses) have completed. Granted, I
> think that gives Paul his read/read ordering, but it's a lot heavier than
> what's required.
I do not believe that we need the trailing "dsb st".
> > And notice how that requires you to mark the store itself. There is no
> > actual barrier *after* the store that does the optimized model.
> >
> > Of course, it's entirely possible that it's not worth worrying about
> > this on ARM64, and that just doing it as a "normal store followed by a
> > full memory barrier" is good enough. But at least in *theory* a
> > microarchitecture might make it much cheaper to do a "store with
> > release consistency" followed by "write barrier".
>
> I agree with the sentiment but, given that this stuff is so heavily
> microarchitecture-dependent (and not simple to probe), a simple dmb ish
> might be the best option after all. That's especially true if the
> microarchitecture decided to ignore the barrier options and treat everything
> as `all accesses, full system' in order to keep the hardware design simple.
I believe that we can do quite a bit better with current hardware
instructions (in the case of ARM, for a recent definition of "current")
and also simplify the memory ordering quite a bit.
Thanx, Paul
^ permalink raw reply
* Re: [RFC] arch: Introduce new TSO memory barrier smp_tmb()
From: Peter Zijlstra @ 2013-11-04 19:11 UTC (permalink / raw)
To: Paul E. McKenney
Cc: Michael Neuling, Mathieu Desnoyers, Oleg Nesterov, LKML,
Linux PPC dev, Anton Blanchard, Frederic Weisbecker,
Victor Kaplansky, Linus Torvalds
In-Reply-To: <20131104162732.GN3947@linux.vnet.ibm.com>
On Mon, Nov 04, 2013 at 08:27:32AM -0800, Paul E. McKenney wrote:
> All this is leading me to suggest the following shortenings of names:
>
> smp_load_with_acquire_semantics() -> smp_load_acquire()
>
> smp_store_with_release_semantics() -> smp_store_release()
>
> But names aside, the above gets rid of explicit barriers on TSO architectures,
> allows ARM to avoid full DMB, and allows PowerPC to use lwsync instead of
> the heavier-weight sync.
A little something like this? Completely guessed at the arm/arm64/ia64
asm, but at least for those archs I found proper instructions (I hope),
for x86,sparc,s390 which are TSO we can do with a barrier and PPC like
said can do with the lwsync, all others fall back to using a smp_mb().
Should probably come with a proper changelog and an addition to _The_
document.
---
arch/alpha/include/asm/barrier.h | 13 +++++++++++
arch/arc/include/asm/barrier.h | 13 +++++++++++
arch/arm/include/asm/barrier.h | 26 +++++++++++++++++++++
arch/arm64/include/asm/barrier.h | 28 +++++++++++++++++++++++
arch/avr32/include/asm/barrier.h | 12 ++++++++++
arch/blackfin/include/asm/barrier.h | 13 +++++++++++
arch/cris/include/asm/barrier.h | 13 +++++++++++
arch/frv/include/asm/barrier.h | 13 +++++++++++
arch/h8300/include/asm/barrier.h | 13 +++++++++++
arch/hexagon/include/asm/barrier.h | 13 +++++++++++
arch/ia64/include/asm/barrier.h | 43 +++++++++++++++++++++++++++++++++++
arch/m32r/include/asm/barrier.h | 13 +++++++++++
arch/m68k/include/asm/barrier.h | 13 +++++++++++
arch/metag/include/asm/barrier.h | 13 +++++++++++
arch/microblaze/include/asm/barrier.h | 13 +++++++++++
arch/mips/include/asm/barrier.h | 13 +++++++++++
arch/mn10300/include/asm/barrier.h | 13 +++++++++++
arch/parisc/include/asm/barrier.h | 13 +++++++++++
arch/powerpc/include/asm/barrier.h | 15 ++++++++++++
arch/s390/include/asm/barrier.h | 13 +++++++++++
arch/score/include/asm/barrier.h | 13 +++++++++++
arch/sh/include/asm/barrier.h | 13 +++++++++++
arch/sparc/include/asm/barrier_32.h | 13 +++++++++++
arch/sparc/include/asm/barrier_64.h | 13 +++++++++++
arch/tile/include/asm/barrier.h | 13 +++++++++++
arch/unicore32/include/asm/barrier.h | 13 +++++++++++
arch/x86/include/asm/barrier.h | 13 +++++++++++
arch/xtensa/include/asm/barrier.h | 13 +++++++++++
28 files changed, 423 insertions(+)
diff --git a/arch/alpha/include/asm/barrier.h b/arch/alpha/include/asm/barrier.h
index ce8860a0b32d..464139feee97 100644
--- a/arch/alpha/include/asm/barrier.h
+++ b/arch/alpha/include/asm/barrier.h
@@ -29,6 +29,19 @@ __asm__ __volatile__("mb": : :"memory")
#define smp_read_barrier_depends() do { } while (0)
#endif
+#define smp_store_release(p, v) \
+do { \
+ smp_mb(); \
+ ACCESS_ONCE(p) = (v); \
+} while (0)
+
+#define smp_load_acquire(p) \
+do { \
+ typeof(p) ___p1 = ACCESS_ONCE(p); \
+ smp_mb(); \
+ return ___p1; \
+} while (0)
+
#define set_mb(var, value) \
do { var = value; mb(); } while (0)
diff --git a/arch/arc/include/asm/barrier.h b/arch/arc/include/asm/barrier.h
index f6cb7c4ffb35..a779da846fb5 100644
--- a/arch/arc/include/asm/barrier.h
+++ b/arch/arc/include/asm/barrier.h
@@ -30,6 +30,19 @@
#define smp_wmb() barrier()
#endif
+#define smp_store_release(p, v) \
+do { \
+ smp_mb(); \
+ ACCESS_ONCE(p) = (v); \
+} while (0)
+
+#define smp_load_acquire(p) \
+do { \
+ typeof(p) ___p1 = ACCESS_ONCE(p); \
+ smp_mb(); \
+ return ___p1; \
+} while (0)
+
#define smp_mb__before_atomic_dec() barrier()
#define smp_mb__after_atomic_dec() barrier()
#define smp_mb__before_atomic_inc() barrier()
diff --git a/arch/arm/include/asm/barrier.h b/arch/arm/include/asm/barrier.h
index 60f15e274e6d..a804093d6891 100644
--- a/arch/arm/include/asm/barrier.h
+++ b/arch/arm/include/asm/barrier.h
@@ -53,10 +53,36 @@
#define smp_mb() barrier()
#define smp_rmb() barrier()
#define smp_wmb() barrier()
+
+#define smp_store_release(p, v) \
+do { \
+ smp_mb(); \
+ ACCESS_ONCE(p) = (v); \
+} while (0)
+
+#define smp_load_acquire(p, v) \
+do { \
+ typeof(p) ___p1 = ACCESS_ONCE(p); \
+ smp_mb(); \
+ return ___p1; \
+} while (0)
#else
#define smp_mb() dmb(ish)
#define smp_rmb() smp_mb()
#define smp_wmb() dmb(ishst)
+
+#define smp_store_release(p, v) \
+do { \
+ asm volatile ("stlr %w0 [%1]" : : "r" (v), "r" (&p) : "memory");\
+} while (0)
+
+#define smp_load_acquire(p) \
+do { \
+ typeof(p) ___p1; \
+ asm volatile ("ldar %w0, [%1]" \
+ : "=r" (___p1) : "r" (&p) : "memory"); \
+ return ___p1; \
+} while (0)
#endif
#define read_barrier_depends() do { } while(0)
diff --git a/arch/arm64/include/asm/barrier.h b/arch/arm64/include/asm/barrier.h
index d4a63338a53c..0da2d4ebb9a8 100644
--- a/arch/arm64/include/asm/barrier.h
+++ b/arch/arm64/include/asm/barrier.h
@@ -35,10 +35,38 @@
#define smp_mb() barrier()
#define smp_rmb() barrier()
#define smp_wmb() barrier()
+
+#define smp_store_release(p, v) \
+do { \
+ smp_mb(); \
+ ACCESS_ONCE(p) = (v); \
+} while (0)
+
+#define smp_load_acquire(p, v) \
+do { \
+ typeof(p) ___p1 = ACCESS_ONCE(p); \
+ smp_mb(); \
+ return ___p1; \
+} while (0)
+
#else
+
#define smp_mb() asm volatile("dmb ish" : : : "memory")
#define smp_rmb() asm volatile("dmb ishld" : : : "memory")
#define smp_wmb() asm volatile("dmb ishst" : : : "memory")
+
+#define smp_store_release(p, v) \
+do { \
+ asm volatile ("stlr %w0 [%1]" : : "r" (v), "r" (&p) : "memory");\
+} while (0)
+
+#define smp_load_acquire(p) \
+do { \
+ typeof(p) ___p1; \
+ asm volatile ("ldar %w0, [%1]" \
+ : "=r" (___p1) : "r" (&p) : "memory"); \
+ return ___p1; \
+} while (0)
#endif
#define read_barrier_depends() do { } while(0)
diff --git a/arch/avr32/include/asm/barrier.h b/arch/avr32/include/asm/barrier.h
index 0961275373db..a0c48ad684f8 100644
--- a/arch/avr32/include/asm/barrier.h
+++ b/arch/avr32/include/asm/barrier.h
@@ -25,5 +25,17 @@
# define smp_read_barrier_depends() do { } while(0)
#endif
+#define smp_store_release(p, v) \
+do { \
+ smp_mb(); \
+ ACCESS_ONCE(p) = (v); \
+} while (0)
+
+#define smp_load_acquire(p, v) \
+do { \
+ typeof(p) ___p1 = ACCESS_ONCE(p); \
+ smp_mb(); \
+ return ___p1; \
+} while (0)
#endif /* __ASM_AVR32_BARRIER_H */
diff --git a/arch/blackfin/include/asm/barrier.h b/arch/blackfin/include/asm/barrier.h
index ebb189507dd7..67889d9225d9 100644
--- a/arch/blackfin/include/asm/barrier.h
+++ b/arch/blackfin/include/asm/barrier.h
@@ -45,4 +45,17 @@
#define set_mb(var, value) do { var = value; mb(); } while (0)
#define smp_read_barrier_depends() read_barrier_depends()
+#define smp_store_release(p, v) \
+do { \
+ smp_mb(); \
+ ACCESS_ONCE(p) = (v); \
+} while (0)
+
+#define smp_load_acquire(p, v) \
+do { \
+ typeof(p) ___p1 = ACCESS_ONCE(p); \
+ smp_mb(); \
+ return ___p1; \
+} while (0)
+
#endif /* _BLACKFIN_BARRIER_H */
diff --git a/arch/cris/include/asm/barrier.h b/arch/cris/include/asm/barrier.h
index 198ad7fa6b25..34243dc44ef1 100644
--- a/arch/cris/include/asm/barrier.h
+++ b/arch/cris/include/asm/barrier.h
@@ -22,4 +22,17 @@
#define smp_read_barrier_depends() do { } while(0)
#endif
+#define smp_store_release(p, v) \
+do { \
+ smp_mb(); \
+ ACCESS_ONCE(p) = (v); \
+} while (0)
+
+#define smp_load_acquire(p, v) \
+do { \
+ typeof(p) ___p1 = ACCESS_ONCE(p); \
+ smp_mb(); \
+ return ___p1; \
+} while (0)
+
#endif /* __ASM_CRIS_BARRIER_H */
diff --git a/arch/frv/include/asm/barrier.h b/arch/frv/include/asm/barrier.h
index 06776ad9f5e9..92f89934d4ed 100644
--- a/arch/frv/include/asm/barrier.h
+++ b/arch/frv/include/asm/barrier.h
@@ -26,4 +26,17 @@
#define set_mb(var, value) \
do { var = (value); barrier(); } while (0)
+#define smp_store_release(p, v) \
+do { \
+ smp_mb(); \
+ ACCESS_ONCE(p) = (v); \
+} while (0)
+
+#define smp_load_acquire(p, v) \
+do { \
+ typeof(p) ___p1 = ACCESS_ONCE(p); \
+ smp_mb(); \
+ return ___p1; \
+} while (0)
+
#endif /* _ASM_BARRIER_H */
diff --git a/arch/h8300/include/asm/barrier.h b/arch/h8300/include/asm/barrier.h
index 9e0aa9fc195d..516e9d379e25 100644
--- a/arch/h8300/include/asm/barrier.h
+++ b/arch/h8300/include/asm/barrier.h
@@ -26,4 +26,17 @@
#define smp_read_barrier_depends() do { } while(0)
#endif
+#define smp_store_release(p, v) \
+do { \
+ smp_mb(); \
+ ACCESS_ONCE(p) = (v); \
+} while (0)
+
+#define smp_load_acquire(p, v) \
+do { \
+ typeof(p) ___p1 = ACCESS_ONCE(p); \
+ smp_mb(); \
+ return ___p1; \
+} while (0)
+
#endif /* _H8300_BARRIER_H */
diff --git a/arch/hexagon/include/asm/barrier.h b/arch/hexagon/include/asm/barrier.h
index 1041a8e70ce8..838a2ebe07a5 100644
--- a/arch/hexagon/include/asm/barrier.h
+++ b/arch/hexagon/include/asm/barrier.h
@@ -38,4 +38,17 @@
#define set_mb(var, value) \
do { var = value; mb(); } while (0)
+#define smp_store_release(p, v) \
+do { \
+ smp_mb(); \
+ ACCESS_ONCE(p) = (v); \
+} while (0)
+
+#define smp_load_acquire(p, v) \
+do { \
+ typeof(p) ___p1 = ACCESS_ONCE(p); \
+ smp_mb(); \
+ return ___p1; \
+} while (0)
+
#endif /* _ASM_BARRIER_H */
diff --git a/arch/ia64/include/asm/barrier.h b/arch/ia64/include/asm/barrier.h
index 60576e06b6fb..4598d390fabb 100644
--- a/arch/ia64/include/asm/barrier.h
+++ b/arch/ia64/include/asm/barrier.h
@@ -45,11 +45,54 @@
# define smp_rmb() rmb()
# define smp_wmb() wmb()
# define smp_read_barrier_depends() read_barrier_depends()
+
+#define smp_store_release(p, v) \
+do { \
+ switch (sizeof(p)) { \
+ case 4: \
+ asm volatile ("st4.acq [%0]=%1" \
+ :: "r" (&p), "r" (v) : "memory"); \
+ break; \
+ case 8: \
+ asm volatile ("st8.acq [%0]=%1" \
+ :: "r" (&p), "r" (v) : "memory"); \
+ break; \
+ } \
+} while (0)
+
+#define smp_load_acquire(p, v) \
+do { \
+ typeof(p) ___p1; \
+ switch (sizeof(p)) { \
+ case 4: \
+ asm volatile ("ld4.rel %0=[%1]" \
+ : "=r"(___p1) : "r" (&p) : "memory"); \
+ break; \
+ case 8: \
+ asm volatile ("ld8.rel %0=[%1]" \
+ : "=r"(___p1) : "r" (&p) : "memory"); \
+ break; \
+ } \
+ return ___p1; \
+} while (0)
#else
# define smp_mb() barrier()
# define smp_rmb() barrier()
# define smp_wmb() barrier()
# define smp_read_barrier_depends() do { } while(0)
+
+#define smp_store_release(p, v) \
+do { \
+ smp_mb(); \
+ ACCESS_ONCE(p) = (v); \
+} while (0)
+
+#define smp_load_acquire(p, v) \
+do { \
+ typeof(p) ___p1 = ACCESS_ONCE(p); \
+ smp_mb(); \
+ return ___p1; \
+} while (0)
#endif
/*
diff --git a/arch/m32r/include/asm/barrier.h b/arch/m32r/include/asm/barrier.h
index 6976621efd3f..e5d42bcf90c5 100644
--- a/arch/m32r/include/asm/barrier.h
+++ b/arch/m32r/include/asm/barrier.h
@@ -91,4 +91,17 @@
#define set_mb(var, value) do { var = value; barrier(); } while (0)
#endif
+#define smp_store_release(p, v) \
+do { \
+ smp_mb(); \
+ ACCESS_ONCE(p) = (v); \
+} while (0)
+
+#define smp_load_acquire(p, v) \
+do { \
+ typeof(p) ___p1 = ACCESS_ONCE(p); \
+ smp_mb(); \
+ return ___p1; \
+} while (0)
+
#endif /* _ASM_M32R_BARRIER_H */
diff --git a/arch/m68k/include/asm/barrier.h b/arch/m68k/include/asm/barrier.h
index 445ce22c23cb..eeb9ecf713cc 100644
--- a/arch/m68k/include/asm/barrier.h
+++ b/arch/m68k/include/asm/barrier.h
@@ -17,4 +17,17 @@
#define smp_wmb() barrier()
#define smp_read_barrier_depends() ((void)0)
+#define smp_store_release(p, v) \
+do { \
+ smp_mb(); \
+ ACCESS_ONCE(p) = (v); \
+} while (0)
+
+#define smp_load_acquire(p, v) \
+do { \
+ typeof(p) ___p1 = ACCESS_ONCE(p); \
+ smp_mb(); \
+ return ___p1; \
+} while (0)
+
#endif /* _M68K_BARRIER_H */
diff --git a/arch/metag/include/asm/barrier.h b/arch/metag/include/asm/barrier.h
index c90bfc6bf648..d8e6f2e4a27c 100644
--- a/arch/metag/include/asm/barrier.h
+++ b/arch/metag/include/asm/barrier.h
@@ -82,4 +82,17 @@ static inline void fence(void)
#define smp_read_barrier_depends() do { } while (0)
#define set_mb(var, value) do { var = value; smp_mb(); } while (0)
+#define smp_store_release(p, v) \
+do { \
+ smp_mb(); \
+ ACCESS_ONCE(p) = (v); \
+} while (0)
+
+#define smp_load_acquire(p, v) \
+do { \
+ typeof(p) ___p1 = ACCESS_ONCE(p); \
+ smp_mb(); \
+ return ___p1; \
+} while (0)
+
#endif /* _ASM_METAG_BARRIER_H */
diff --git a/arch/microblaze/include/asm/barrier.h b/arch/microblaze/include/asm/barrier.h
index df5be3e87044..a890702061c9 100644
--- a/arch/microblaze/include/asm/barrier.h
+++ b/arch/microblaze/include/asm/barrier.h
@@ -24,4 +24,17 @@
#define smp_rmb() rmb()
#define smp_wmb() wmb()
+#define smp_store_release(p, v) \
+do { \
+ smp_mb(); \
+ ACCESS_ONCE(p) = (v); \
+} while (0)
+
+#define smp_load_acquire(p, v) \
+do { \
+ typeof(p) ___p1 = ACCESS_ONCE(p); \
+ smp_mb(); \
+ return ___p1; \
+} while (0)
+
#endif /* _ASM_MICROBLAZE_BARRIER_H */
diff --git a/arch/mips/include/asm/barrier.h b/arch/mips/include/asm/barrier.h
index 314ab5532019..e59bcd051f36 100644
--- a/arch/mips/include/asm/barrier.h
+++ b/arch/mips/include/asm/barrier.h
@@ -180,4 +180,17 @@
#define nudge_writes() mb()
#endif
+#define smp_store_release(p, v) \
+do { \
+ smp_mb(); \
+ ACCESS_ONCE(p) = (v); \
+} while (0)
+
+#define smp_load_acquire(p, v) \
+do { \
+ typeof(p) ___p1 = ACCESS_ONCE(p); \
+ smp_mb(); \
+ return ___p1; \
+} while (0)
+
#endif /* __ASM_BARRIER_H */
diff --git a/arch/mn10300/include/asm/barrier.h b/arch/mn10300/include/asm/barrier.h
index 2bd97a5c8af7..0e6a0608d4a1 100644
--- a/arch/mn10300/include/asm/barrier.h
+++ b/arch/mn10300/include/asm/barrier.h
@@ -34,4 +34,17 @@
#define read_barrier_depends() do {} while (0)
#define smp_read_barrier_depends() do {} while (0)
+#define smp_store_release(p, v) \
+do { \
+ smp_mb(); \
+ ACCESS_ONCE(p) = (v); \
+} while (0)
+
+#define smp_load_acquire(p, v) \
+do { \
+ typeof(p) ___p1 = ACCESS_ONCE(p); \
+ smp_mb(); \
+ return ___p1; \
+} while (0)
+
#endif /* _ASM_BARRIER_H */
diff --git a/arch/parisc/include/asm/barrier.h b/arch/parisc/include/asm/barrier.h
index e77d834aa803..f1145a8594a0 100644
--- a/arch/parisc/include/asm/barrier.h
+++ b/arch/parisc/include/asm/barrier.h
@@ -32,4 +32,17 @@
#define set_mb(var, value) do { var = value; mb(); } while (0)
+#define smp_store_release(p, v) \
+do { \
+ smp_mb(); \
+ ACCESS_ONCE(p) = (v); \
+} while (0)
+
+#define smp_load_acquire(p, v) \
+do { \
+ typeof(p) ___p1 = ACCESS_ONCE(p); \
+ smp_mb(); \
+ return ___p1; \
+} while (0)
+
#endif /* __PARISC_BARRIER_H */
diff --git a/arch/powerpc/include/asm/barrier.h b/arch/powerpc/include/asm/barrier.h
index ae782254e731..b5cc36791f42 100644
--- a/arch/powerpc/include/asm/barrier.h
+++ b/arch/powerpc/include/asm/barrier.h
@@ -65,4 +65,19 @@
#define data_barrier(x) \
asm volatile("twi 0,%0,0; isync" : : "r" (x) : "memory");
+/* use smp_rmb() as that is either lwsync or a barrier() depending on SMP */
+
+#define smp_store_release(p, v) \
+do { \
+ smp_rmb(); \
+ ACCESS_ONCE(p) = (v); \
+} while (0)
+
+#define smp_load_acquire(p, v) \
+do { \
+ typeof(p) ___p1 = ACCESS_ONCE(p); \
+ smp_rmb(); \
+ return ___p1; \
+} while (0)
+
#endif /* _ASM_POWERPC_BARRIER_H */
diff --git a/arch/s390/include/asm/barrier.h b/arch/s390/include/asm/barrier.h
index 16760eeb79b0..e8989c40e11c 100644
--- a/arch/s390/include/asm/barrier.h
+++ b/arch/s390/include/asm/barrier.h
@@ -32,4 +32,17 @@
#define set_mb(var, value) do { var = value; mb(); } while (0)
+#define smp_store_release(p, v) \
+do { \
+ barrier(); \
+ ACCESS_ONCE(p) = (v); \
+} while (0)
+
+#define smp_load_acquire(p, v) \
+do { \
+ typeof(p) ___p1 = ACCESS_ONCE(p); \
+ barrier(); \
+ return ___p1; \
+} while (0)
+
#endif /* __ASM_BARRIER_H */
diff --git a/arch/score/include/asm/barrier.h b/arch/score/include/asm/barrier.h
index 0eacb6471e6d..5f101ef8ade9 100644
--- a/arch/score/include/asm/barrier.h
+++ b/arch/score/include/asm/barrier.h
@@ -13,4 +13,17 @@
#define set_mb(var, value) do {var = value; wmb(); } while (0)
+#define smp_store_release(p, v) \
+do { \
+ smp_mb(); \
+ ACCESS_ONCE(p) = (v); \
+} while (0)
+
+#define smp_load_acquire(p, v) \
+do { \
+ typeof(p) ___p1 = ACCESS_ONCE(p); \
+ smp_mb(); \
+ return ___p1; \
+} while (0)
+
#endif /* _ASM_SCORE_BARRIER_H */
diff --git a/arch/sh/include/asm/barrier.h b/arch/sh/include/asm/barrier.h
index 72c103dae300..611128c2f636 100644
--- a/arch/sh/include/asm/barrier.h
+++ b/arch/sh/include/asm/barrier.h
@@ -51,4 +51,17 @@
#define set_mb(var, value) do { (void)xchg(&var, value); } while (0)
+#define smp_store_release(p, v) \
+do { \
+ smp_mb(); \
+ ACCESS_ONCE(p) = (v); \
+} while (0)
+
+#define smp_load_acquire(p, v) \
+do { \
+ typeof(p) ___p1 = ACCESS_ONCE(p); \
+ smp_mb(); \
+ return ___p1; \
+} while (0)
+
#endif /* __ASM_SH_BARRIER_H */
diff --git a/arch/sparc/include/asm/barrier_32.h b/arch/sparc/include/asm/barrier_32.h
index c1b76654ee76..f47f9d51f326 100644
--- a/arch/sparc/include/asm/barrier_32.h
+++ b/arch/sparc/include/asm/barrier_32.h
@@ -12,4 +12,17 @@
#define smp_wmb() __asm__ __volatile__("":::"memory")
#define smp_read_barrier_depends() do { } while(0)
+#define smp_store_release(p, v) \
+do { \
+ smp_mb(); \
+ ACCESS_ONCE(p) = (v); \
+} while (0)
+
+#define smp_load_acquire(p, v) \
+do { \
+ typeof(p) ___p1 = ACCESS_ONCE(p); \
+ smp_mb(); \
+ return ___p1; \
+} while (0)
+
#endif /* !(__SPARC_BARRIER_H) */
diff --git a/arch/sparc/include/asm/barrier_64.h b/arch/sparc/include/asm/barrier_64.h
index 95d45986f908..77cbe6982ca0 100644
--- a/arch/sparc/include/asm/barrier_64.h
+++ b/arch/sparc/include/asm/barrier_64.h
@@ -53,4 +53,17 @@ do { __asm__ __volatile__("ba,pt %%xcc, 1f\n\t" \
#define smp_read_barrier_depends() do { } while(0)
+#define smp_store_release(p, v) \
+do { \
+ barrier(); \
+ ACCESS_ONCE(p) = (v); \
+} while (0)
+
+#define smp_load_acquire(p, v) \
+do { \
+ typeof(p) ___p1 = ACCESS_ONCE(p); \
+ barrier(); \
+ return ___p1; \
+} while (0)
+
#endif /* !(__SPARC64_BARRIER_H) */
diff --git a/arch/tile/include/asm/barrier.h b/arch/tile/include/asm/barrier.h
index a9a73da5865d..4d5330d4fd31 100644
--- a/arch/tile/include/asm/barrier.h
+++ b/arch/tile/include/asm/barrier.h
@@ -140,5 +140,18 @@ mb_incoherent(void)
#define set_mb(var, value) \
do { var = value; mb(); } while (0)
+#define smp_store_release(p, v) \
+do { \
+ smp_mb(); \
+ ACCESS_ONCE(p) = (v); \
+} while (0)
+
+#define smp_load_acquire(p, v) \
+do { \
+ typeof(p) ___p1 = ACCESS_ONCE(p); \
+ smp_mb(); \
+ return ___p1; \
+} while (0)
+
#endif /* !__ASSEMBLY__ */
#endif /* _ASM_TILE_BARRIER_H */
diff --git a/arch/unicore32/include/asm/barrier.h b/arch/unicore32/include/asm/barrier.h
index a6620e5336b6..5471ff6aae10 100644
--- a/arch/unicore32/include/asm/barrier.h
+++ b/arch/unicore32/include/asm/barrier.h
@@ -25,4 +25,17 @@
#define set_mb(var, value) do { var = value; smp_mb(); } while (0)
+#define smp_store_release(p, v) \
+do { \
+ smp_mb(); \
+ ACCESS_ONCE(p) = (v); \
+} while (0)
+
+#define smp_load_acquire(p, v) \
+do { \
+ typeof(p) ___p1 = ACCESS_ONCE(p); \
+ smp_mb(); \
+ return ___p1; \
+} while (0)
+
#endif /* __UNICORE_BARRIER_H__ */
diff --git a/arch/x86/include/asm/barrier.h b/arch/x86/include/asm/barrier.h
index c6cd358a1eec..a7fd8201ab09 100644
--- a/arch/x86/include/asm/barrier.h
+++ b/arch/x86/include/asm/barrier.h
@@ -100,6 +100,19 @@
#define set_mb(var, value) do { var = value; barrier(); } while (0)
#endif
+#define smp_store_release(p, v) \
+do { \
+ barrier(); \
+ ACCESS_ONCE(p) = (v); \
+} while (0)
+
+#define smp_load_acquire(p, v) \
+do { \
+ typeof(p) ___p1 = ACCESS_ONCE(p); \
+ barrier(); \
+ return ___p1; \
+} while (0)
+
/*
* Stop RDTSC speculation. This is needed when you need to use RDTSC
* (or get_cycles or vread that possibly accesses the TSC) in a defined
diff --git a/arch/xtensa/include/asm/barrier.h b/arch/xtensa/include/asm/barrier.h
index ef021677d536..703d511add49 100644
--- a/arch/xtensa/include/asm/barrier.h
+++ b/arch/xtensa/include/asm/barrier.h
@@ -26,4 +26,17 @@
#define set_mb(var, value) do { var = value; mb(); } while (0)
+#define smp_store_release(p, v) \
+do { \
+ smp_mb(); \
+ ACCESS_ONCE(p) = (v); \
+} while (0)
+
+#define smp_load_acquire(p, v) \
+do { \
+ typeof(p) ___p1 = ACCESS_ONCE(p); \
+ smp_mb(); \
+ return ___p1; \
+} while (0)
+
#endif /* _XTENSA_SYSTEM_H */
^ permalink raw reply related
* Re: [RFC] arch: Introduce new TSO memory barrier smp_tmb()
From: Peter Zijlstra @ 2013-11-04 19:18 UTC (permalink / raw)
To: Paul E. McKenney
Cc: Michael Neuling, Mathieu Desnoyers, Oleg Nesterov, LKML,
Linux PPC dev, Anton Blanchard, Frederic Weisbecker,
Victor Kaplansky, Linus Torvalds
In-Reply-To: <20131104191127.GW16117@laptop.programming.kicks-ass.net>
On Mon, Nov 04, 2013 at 08:11:27PM +0100, Peter Zijlstra wrote:
> +#define smp_load_acquire(p, v) \
I R idiot!! :-)
---
arch/alpha/include/asm/barrier.h | 13 +++++++++++
arch/arc/include/asm/barrier.h | 13 +++++++++++
arch/arm/include/asm/barrier.h | 26 +++++++++++++++++++++
arch/arm64/include/asm/barrier.h | 28 +++++++++++++++++++++++
arch/avr32/include/asm/barrier.h | 12 ++++++++++
arch/blackfin/include/asm/barrier.h | 13 +++++++++++
arch/cris/include/asm/barrier.h | 13 +++++++++++
arch/frv/include/asm/barrier.h | 13 +++++++++++
arch/h8300/include/asm/barrier.h | 13 +++++++++++
arch/hexagon/include/asm/barrier.h | 13 +++++++++++
arch/ia64/include/asm/barrier.h | 43 +++++++++++++++++++++++++++++++++++
arch/m32r/include/asm/barrier.h | 13 +++++++++++
arch/m68k/include/asm/barrier.h | 13 +++++++++++
arch/metag/include/asm/barrier.h | 13 +++++++++++
arch/microblaze/include/asm/barrier.h | 13 +++++++++++
arch/mips/include/asm/barrier.h | 13 +++++++++++
arch/mn10300/include/asm/barrier.h | 13 +++++++++++
arch/parisc/include/asm/barrier.h | 13 +++++++++++
arch/powerpc/include/asm/barrier.h | 15 ++++++++++++
arch/s390/include/asm/barrier.h | 13 +++++++++++
arch/score/include/asm/barrier.h | 13 +++++++++++
arch/sh/include/asm/barrier.h | 13 +++++++++++
arch/sparc/include/asm/barrier_32.h | 13 +++++++++++
arch/sparc/include/asm/barrier_64.h | 13 +++++++++++
arch/tile/include/asm/barrier.h | 13 +++++++++++
arch/unicore32/include/asm/barrier.h | 13 +++++++++++
arch/x86/include/asm/barrier.h | 13 +++++++++++
arch/xtensa/include/asm/barrier.h | 13 +++++++++++
28 files changed, 423 insertions(+)
diff --git a/arch/alpha/include/asm/barrier.h b/arch/alpha/include/asm/barrier.h
index ce8860a0b32d..464139feee97 100644
--- a/arch/alpha/include/asm/barrier.h
+++ b/arch/alpha/include/asm/barrier.h
@@ -29,6 +29,19 @@ __asm__ __volatile__("mb": : :"memory")
#define smp_read_barrier_depends() do { } while (0)
#endif
+#define smp_store_release(p, v) \
+do { \
+ smp_mb(); \
+ ACCESS_ONCE(p) = (v); \
+} while (0)
+
+#define smp_load_acquire(p) \
+do { \
+ typeof(p) ___p1 = ACCESS_ONCE(p); \
+ smp_mb(); \
+ return ___p1; \
+} while (0)
+
#define set_mb(var, value) \
do { var = value; mb(); } while (0)
diff --git a/arch/arc/include/asm/barrier.h b/arch/arc/include/asm/barrier.h
index f6cb7c4ffb35..a779da846fb5 100644
--- a/arch/arc/include/asm/barrier.h
+++ b/arch/arc/include/asm/barrier.h
@@ -30,6 +30,19 @@
#define smp_wmb() barrier()
#endif
+#define smp_store_release(p, v) \
+do { \
+ smp_mb(); \
+ ACCESS_ONCE(p) = (v); \
+} while (0)
+
+#define smp_load_acquire(p) \
+do { \
+ typeof(p) ___p1 = ACCESS_ONCE(p); \
+ smp_mb(); \
+ return ___p1; \
+} while (0)
+
#define smp_mb__before_atomic_dec() barrier()
#define smp_mb__after_atomic_dec() barrier()
#define smp_mb__before_atomic_inc() barrier()
diff --git a/arch/arm/include/asm/barrier.h b/arch/arm/include/asm/barrier.h
index 60f15e274e6d..4ada4720bdeb 100644
--- a/arch/arm/include/asm/barrier.h
+++ b/arch/arm/include/asm/barrier.h
@@ -53,10 +53,36 @@
#define smp_mb() barrier()
#define smp_rmb() barrier()
#define smp_wmb() barrier()
+
+#define smp_store_release(p, v) \
+do { \
+ smp_mb(); \
+ ACCESS_ONCE(p) = (v); \
+} while (0)
+
+#define smp_load_acquire(p) \
+do { \
+ typeof(p) ___p1 = ACCESS_ONCE(p); \
+ smp_mb(); \
+ return ___p1; \
+} while (0)
#else
#define smp_mb() dmb(ish)
#define smp_rmb() smp_mb()
#define smp_wmb() dmb(ishst)
+
+#define smp_store_release(p, v) \
+do { \
+ asm volatile ("stlr %w0 [%1]" : : "r" (v), "r" (&p) : "memory");\
+} while (0)
+
+#define smp_load_acquire(p) \
+do { \
+ typeof(p) ___p1; \
+ asm volatile ("ldar %w0, [%1]" \
+ : "=r" (___p1) : "r" (&p) : "memory"); \
+ return ___p1; \
+} while (0)
#endif
#define read_barrier_depends() do { } while(0)
diff --git a/arch/arm64/include/asm/barrier.h b/arch/arm64/include/asm/barrier.h
index d4a63338a53c..3dfddc0416f6 100644
--- a/arch/arm64/include/asm/barrier.h
+++ b/arch/arm64/include/asm/barrier.h
@@ -35,10 +35,38 @@
#define smp_mb() barrier()
#define smp_rmb() barrier()
#define smp_wmb() barrier()
+
+#define smp_store_release(p, v) \
+do { \
+ smp_mb(); \
+ ACCESS_ONCE(p) = (v); \
+} while (0)
+
+#define smp_load_acquire(p) \
+do { \
+ typeof(p) ___p1 = ACCESS_ONCE(p); \
+ smp_mb(); \
+ return ___p1; \
+} while (0)
+
#else
+
#define smp_mb() asm volatile("dmb ish" : : : "memory")
#define smp_rmb() asm volatile("dmb ishld" : : : "memory")
#define smp_wmb() asm volatile("dmb ishst" : : : "memory")
+
+#define smp_store_release(p, v) \
+do { \
+ asm volatile ("stlr %w0 [%1]" : : "r" (v), "r" (&p) : "memory");\
+} while (0)
+
+#define smp_load_acquire(p) \
+do { \
+ typeof(p) ___p1; \
+ asm volatile ("ldar %w0, [%1]" \
+ : "=r" (___p1) : "r" (&p) : "memory"); \
+ return ___p1; \
+} while (0)
#endif
#define read_barrier_depends() do { } while(0)
diff --git a/arch/avr32/include/asm/barrier.h b/arch/avr32/include/asm/barrier.h
index 0961275373db..8fd164648e71 100644
--- a/arch/avr32/include/asm/barrier.h
+++ b/arch/avr32/include/asm/barrier.h
@@ -25,5 +25,17 @@
# define smp_read_barrier_depends() do { } while(0)
#endif
+#define smp_store_release(p, v) \
+do { \
+ smp_mb(); \
+ ACCESS_ONCE(p) = (v); \
+} while (0)
+
+#define smp_load_acquire(p) \
+do { \
+ typeof(p) ___p1 = ACCESS_ONCE(p); \
+ smp_mb(); \
+ return ___p1; \
+} while (0)
#endif /* __ASM_AVR32_BARRIER_H */
diff --git a/arch/blackfin/include/asm/barrier.h b/arch/blackfin/include/asm/barrier.h
index ebb189507dd7..c8b85bba843f 100644
--- a/arch/blackfin/include/asm/barrier.h
+++ b/arch/blackfin/include/asm/barrier.h
@@ -45,4 +45,17 @@
#define set_mb(var, value) do { var = value; mb(); } while (0)
#define smp_read_barrier_depends() read_barrier_depends()
+#define smp_store_release(p, v) \
+do { \
+ smp_mb(); \
+ ACCESS_ONCE(p) = (v); \
+} while (0)
+
+#define smp_load_acquire(p) \
+do { \
+ typeof(p) ___p1 = ACCESS_ONCE(p); \
+ smp_mb(); \
+ return ___p1; \
+} while (0)
+
#endif /* _BLACKFIN_BARRIER_H */
diff --git a/arch/cris/include/asm/barrier.h b/arch/cris/include/asm/barrier.h
index 198ad7fa6b25..26f21f5d1d15 100644
--- a/arch/cris/include/asm/barrier.h
+++ b/arch/cris/include/asm/barrier.h
@@ -22,4 +22,17 @@
#define smp_read_barrier_depends() do { } while(0)
#endif
+#define smp_store_release(p, v) \
+do { \
+ smp_mb(); \
+ ACCESS_ONCE(p) = (v); \
+} while (0)
+
+#define smp_load_acquire(p) \
+do { \
+ typeof(p) ___p1 = ACCESS_ONCE(p); \
+ smp_mb(); \
+ return ___p1; \
+} while (0)
+
#endif /* __ASM_CRIS_BARRIER_H */
diff --git a/arch/frv/include/asm/barrier.h b/arch/frv/include/asm/barrier.h
index 06776ad9f5e9..4569028382fa 100644
--- a/arch/frv/include/asm/barrier.h
+++ b/arch/frv/include/asm/barrier.h
@@ -26,4 +26,17 @@
#define set_mb(var, value) \
do { var = (value); barrier(); } while (0)
+#define smp_store_release(p, v) \
+do { \
+ smp_mb(); \
+ ACCESS_ONCE(p) = (v); \
+} while (0)
+
+#define smp_load_acquire(p) \
+do { \
+ typeof(p) ___p1 = ACCESS_ONCE(p); \
+ smp_mb(); \
+ return ___p1; \
+} while (0)
+
#endif /* _ASM_BARRIER_H */
diff --git a/arch/h8300/include/asm/barrier.h b/arch/h8300/include/asm/barrier.h
index 9e0aa9fc195d..45d36738814d 100644
--- a/arch/h8300/include/asm/barrier.h
+++ b/arch/h8300/include/asm/barrier.h
@@ -26,4 +26,17 @@
#define smp_read_barrier_depends() do { } while(0)
#endif
+#define smp_store_release(p, v) \
+do { \
+ smp_mb(); \
+ ACCESS_ONCE(p) = (v); \
+} while (0)
+
+#define smp_load_acquire(p) \
+do { \
+ typeof(p) ___p1 = ACCESS_ONCE(p); \
+ smp_mb(); \
+ return ___p1; \
+} while (0)
+
#endif /* _H8300_BARRIER_H */
diff --git a/arch/hexagon/include/asm/barrier.h b/arch/hexagon/include/asm/barrier.h
index 1041a8e70ce8..d88d54bd2e6e 100644
--- a/arch/hexagon/include/asm/barrier.h
+++ b/arch/hexagon/include/asm/barrier.h
@@ -38,4 +38,17 @@
#define set_mb(var, value) \
do { var = value; mb(); } while (0)
+#define smp_store_release(p, v) \
+do { \
+ smp_mb(); \
+ ACCESS_ONCE(p) = (v); \
+} while (0)
+
+#define smp_load_acquire(p) \
+do { \
+ typeof(p) ___p1 = ACCESS_ONCE(p); \
+ smp_mb(); \
+ return ___p1; \
+} while (0)
+
#endif /* _ASM_BARRIER_H */
diff --git a/arch/ia64/include/asm/barrier.h b/arch/ia64/include/asm/barrier.h
index 60576e06b6fb..b7f1a8aa03af 100644
--- a/arch/ia64/include/asm/barrier.h
+++ b/arch/ia64/include/asm/barrier.h
@@ -45,11 +45,54 @@
# define smp_rmb() rmb()
# define smp_wmb() wmb()
# define smp_read_barrier_depends() read_barrier_depends()
+
+#define smp_store_release(p, v) \
+do { \
+ switch (sizeof(p)) { \
+ case 4: \
+ asm volatile ("st4.acq [%0]=%1" \
+ :: "r" (&p), "r" (v) : "memory"); \
+ break; \
+ case 8: \
+ asm volatile ("st8.acq [%0]=%1" \
+ :: "r" (&p), "r" (v) : "memory"); \
+ break; \
+ } \
+} while (0)
+
+#define smp_load_acquire(p) \
+do { \
+ typeof(p) ___p1; \
+ switch (sizeof(p)) { \
+ case 4: \
+ asm volatile ("ld4.rel %0=[%1]" \
+ : "=r"(___p1) : "r" (&p) : "memory"); \
+ break; \
+ case 8: \
+ asm volatile ("ld8.rel %0=[%1]" \
+ : "=r"(___p1) : "r" (&p) : "memory"); \
+ break; \
+ } \
+ return ___p1; \
+} while (0)
#else
# define smp_mb() barrier()
# define smp_rmb() barrier()
# define smp_wmb() barrier()
# define smp_read_barrier_depends() do { } while(0)
+
+#define smp_store_release(p, v) \
+do { \
+ smp_mb(); \
+ ACCESS_ONCE(p) = (v); \
+} while (0)
+
+#define smp_load_acquire(p) \
+do { \
+ typeof(p) ___p1 = ACCESS_ONCE(p); \
+ smp_mb(); \
+ return ___p1; \
+} while (0)
#endif
/*
diff --git a/arch/m32r/include/asm/barrier.h b/arch/m32r/include/asm/barrier.h
index 6976621efd3f..d78612289cb2 100644
--- a/arch/m32r/include/asm/barrier.h
+++ b/arch/m32r/include/asm/barrier.h
@@ -91,4 +91,17 @@
#define set_mb(var, value) do { var = value; barrier(); } while (0)
#endif
+#define smp_store_release(p, v) \
+do { \
+ smp_mb(); \
+ ACCESS_ONCE(p) = (v); \
+} while (0)
+
+#define smp_load_acquire(p) \
+do { \
+ typeof(p) ___p1 = ACCESS_ONCE(p); \
+ smp_mb(); \
+ return ___p1; \
+} while (0)
+
#endif /* _ASM_M32R_BARRIER_H */
diff --git a/arch/m68k/include/asm/barrier.h b/arch/m68k/include/asm/barrier.h
index 445ce22c23cb..1e63b11c424c 100644
--- a/arch/m68k/include/asm/barrier.h
+++ b/arch/m68k/include/asm/barrier.h
@@ -17,4 +17,17 @@
#define smp_wmb() barrier()
#define smp_read_barrier_depends() ((void)0)
+#define smp_store_release(p, v) \
+do { \
+ smp_mb(); \
+ ACCESS_ONCE(p) = (v); \
+} while (0)
+
+#define smp_load_acquire(p) \
+do { \
+ typeof(p) ___p1 = ACCESS_ONCE(p); \
+ smp_mb(); \
+ return ___p1; \
+} while (0)
+
#endif /* _M68K_BARRIER_H */
diff --git a/arch/metag/include/asm/barrier.h b/arch/metag/include/asm/barrier.h
index c90bfc6bf648..9ffd0b167f07 100644
--- a/arch/metag/include/asm/barrier.h
+++ b/arch/metag/include/asm/barrier.h
@@ -82,4 +82,17 @@ static inline void fence(void)
#define smp_read_barrier_depends() do { } while (0)
#define set_mb(var, value) do { var = value; smp_mb(); } while (0)
+#define smp_store_release(p, v) \
+do { \
+ smp_mb(); \
+ ACCESS_ONCE(p) = (v); \
+} while (0)
+
+#define smp_load_acquire(p) \
+do { \
+ typeof(p) ___p1 = ACCESS_ONCE(p); \
+ smp_mb(); \
+ return ___p1; \
+} while (0)
+
#endif /* _ASM_METAG_BARRIER_H */
diff --git a/arch/microblaze/include/asm/barrier.h b/arch/microblaze/include/asm/barrier.h
index df5be3e87044..db0b5e205ce3 100644
--- a/arch/microblaze/include/asm/barrier.h
+++ b/arch/microblaze/include/asm/barrier.h
@@ -24,4 +24,17 @@
#define smp_rmb() rmb()
#define smp_wmb() wmb()
+#define smp_store_release(p, v) \
+do { \
+ smp_mb(); \
+ ACCESS_ONCE(p) = (v); \
+} while (0)
+
+#define smp_load_acquire(p) \
+do { \
+ typeof(p) ___p1 = ACCESS_ONCE(p); \
+ smp_mb(); \
+ return ___p1; \
+} while (0)
+
#endif /* _ASM_MICROBLAZE_BARRIER_H */
diff --git a/arch/mips/include/asm/barrier.h b/arch/mips/include/asm/barrier.h
index 314ab5532019..8031afcc7f64 100644
--- a/arch/mips/include/asm/barrier.h
+++ b/arch/mips/include/asm/barrier.h
@@ -180,4 +180,17 @@
#define nudge_writes() mb()
#endif
+#define smp_store_release(p, v) \
+do { \
+ smp_mb(); \
+ ACCESS_ONCE(p) = (v); \
+} while (0)
+
+#define smp_load_acquire(p) \
+do { \
+ typeof(p) ___p1 = ACCESS_ONCE(p); \
+ smp_mb(); \
+ return ___p1; \
+} while (0)
+
#endif /* __ASM_BARRIER_H */
diff --git a/arch/mn10300/include/asm/barrier.h b/arch/mn10300/include/asm/barrier.h
index 2bd97a5c8af7..e822ff76f498 100644
--- a/arch/mn10300/include/asm/barrier.h
+++ b/arch/mn10300/include/asm/barrier.h
@@ -34,4 +34,17 @@
#define read_barrier_depends() do {} while (0)
#define smp_read_barrier_depends() do {} while (0)
+#define smp_store_release(p, v) \
+do { \
+ smp_mb(); \
+ ACCESS_ONCE(p) = (v); \
+} while (0)
+
+#define smp_load_acquire(p) \
+do { \
+ typeof(p) ___p1 = ACCESS_ONCE(p); \
+ smp_mb(); \
+ return ___p1; \
+} while (0)
+
#endif /* _ASM_BARRIER_H */
diff --git a/arch/parisc/include/asm/barrier.h b/arch/parisc/include/asm/barrier.h
index e77d834aa803..58757747f873 100644
--- a/arch/parisc/include/asm/barrier.h
+++ b/arch/parisc/include/asm/barrier.h
@@ -32,4 +32,17 @@
#define set_mb(var, value) do { var = value; mb(); } while (0)
+#define smp_store_release(p, v) \
+do { \
+ smp_mb(); \
+ ACCESS_ONCE(p) = (v); \
+} while (0)
+
+#define smp_load_acquire(p) \
+do { \
+ typeof(p) ___p1 = ACCESS_ONCE(p); \
+ smp_mb(); \
+ return ___p1; \
+} while (0)
+
#endif /* __PARISC_BARRIER_H */
diff --git a/arch/powerpc/include/asm/barrier.h b/arch/powerpc/include/asm/barrier.h
index ae782254e731..54922626b356 100644
--- a/arch/powerpc/include/asm/barrier.h
+++ b/arch/powerpc/include/asm/barrier.h
@@ -65,4 +65,19 @@
#define data_barrier(x) \
asm volatile("twi 0,%0,0; isync" : : "r" (x) : "memory");
+/* use smp_rmb() as that is either lwsync or a barrier() depending on SMP */
+
+#define smp_store_release(p, v) \
+do { \
+ smp_rmb(); \
+ ACCESS_ONCE(p) = (v); \
+} while (0)
+
+#define smp_load_acquire(p) \
+do { \
+ typeof(p) ___p1 = ACCESS_ONCE(p); \
+ smp_rmb(); \
+ return ___p1; \
+} while (0)
+
#endif /* _ASM_POWERPC_BARRIER_H */
diff --git a/arch/s390/include/asm/barrier.h b/arch/s390/include/asm/barrier.h
index 16760eeb79b0..babf928649a4 100644
--- a/arch/s390/include/asm/barrier.h
+++ b/arch/s390/include/asm/barrier.h
@@ -32,4 +32,17 @@
#define set_mb(var, value) do { var = value; mb(); } while (0)
+#define smp_store_release(p, v) \
+do { \
+ barrier(); \
+ ACCESS_ONCE(p) = (v); \
+} while (0)
+
+#define smp_load_acquire(p) \
+do { \
+ typeof(p) ___p1 = ACCESS_ONCE(p); \
+ barrier(); \
+ return ___p1; \
+} while (0)
+
#endif /* __ASM_BARRIER_H */
diff --git a/arch/score/include/asm/barrier.h b/arch/score/include/asm/barrier.h
index 0eacb6471e6d..5905ea57a104 100644
--- a/arch/score/include/asm/barrier.h
+++ b/arch/score/include/asm/barrier.h
@@ -13,4 +13,17 @@
#define set_mb(var, value) do {var = value; wmb(); } while (0)
+#define smp_store_release(p, v) \
+do { \
+ smp_mb(); \
+ ACCESS_ONCE(p) = (v); \
+} while (0)
+
+#define smp_load_acquire(p) \
+do { \
+ typeof(p) ___p1 = ACCESS_ONCE(p); \
+ smp_mb(); \
+ return ___p1; \
+} while (0)
+
#endif /* _ASM_SCORE_BARRIER_H */
diff --git a/arch/sh/include/asm/barrier.h b/arch/sh/include/asm/barrier.h
index 72c103dae300..379f500023b6 100644
--- a/arch/sh/include/asm/barrier.h
+++ b/arch/sh/include/asm/barrier.h
@@ -51,4 +51,17 @@
#define set_mb(var, value) do { (void)xchg(&var, value); } while (0)
+#define smp_store_release(p, v) \
+do { \
+ smp_mb(); \
+ ACCESS_ONCE(p) = (v); \
+} while (0)
+
+#define smp_load_acquire(p) \
+do { \
+ typeof(p) ___p1 = ACCESS_ONCE(p); \
+ smp_mb(); \
+ return ___p1; \
+} while (0)
+
#endif /* __ASM_SH_BARRIER_H */
diff --git a/arch/sparc/include/asm/barrier_32.h b/arch/sparc/include/asm/barrier_32.h
index c1b76654ee76..1649081d1b86 100644
--- a/arch/sparc/include/asm/barrier_32.h
+++ b/arch/sparc/include/asm/barrier_32.h
@@ -12,4 +12,17 @@
#define smp_wmb() __asm__ __volatile__("":::"memory")
#define smp_read_barrier_depends() do { } while(0)
+#define smp_store_release(p, v) \
+do { \
+ smp_mb(); \
+ ACCESS_ONCE(p) = (v); \
+} while (0)
+
+#define smp_load_acquire(p) \
+do { \
+ typeof(p) ___p1 = ACCESS_ONCE(p); \
+ smp_mb(); \
+ return ___p1; \
+} while (0)
+
#endif /* !(__SPARC_BARRIER_H) */
diff --git a/arch/sparc/include/asm/barrier_64.h b/arch/sparc/include/asm/barrier_64.h
index 95d45986f908..5e23ced0a29a 100644
--- a/arch/sparc/include/asm/barrier_64.h
+++ b/arch/sparc/include/asm/barrier_64.h
@@ -53,4 +53,17 @@ do { __asm__ __volatile__("ba,pt %%xcc, 1f\n\t" \
#define smp_read_barrier_depends() do { } while(0)
+#define smp_store_release(p, v) \
+do { \
+ barrier(); \
+ ACCESS_ONCE(p) = (v); \
+} while (0)
+
+#define smp_load_acquire(p) \
+do { \
+ typeof(p) ___p1 = ACCESS_ONCE(p); \
+ barrier(); \
+ return ___p1; \
+} while (0)
+
#endif /* !(__SPARC64_BARRIER_H) */
diff --git a/arch/tile/include/asm/barrier.h b/arch/tile/include/asm/barrier.h
index a9a73da5865d..1f08318db3c0 100644
--- a/arch/tile/include/asm/barrier.h
+++ b/arch/tile/include/asm/barrier.h
@@ -140,5 +140,18 @@ mb_incoherent(void)
#define set_mb(var, value) \
do { var = value; mb(); } while (0)
+#define smp_store_release(p, v) \
+do { \
+ smp_mb(); \
+ ACCESS_ONCE(p) = (v); \
+} while (0)
+
+#define smp_load_acquire(p) \
+do { \
+ typeof(p) ___p1 = ACCESS_ONCE(p); \
+ smp_mb(); \
+ return ___p1; \
+} while (0)
+
#endif /* !__ASSEMBLY__ */
#endif /* _ASM_TILE_BARRIER_H */
diff --git a/arch/unicore32/include/asm/barrier.h b/arch/unicore32/include/asm/barrier.h
index a6620e5336b6..fa8bf69d9a09 100644
--- a/arch/unicore32/include/asm/barrier.h
+++ b/arch/unicore32/include/asm/barrier.h
@@ -25,4 +25,17 @@
#define set_mb(var, value) do { var = value; smp_mb(); } while (0)
+#define smp_store_release(p, v) \
+do { \
+ smp_mb(); \
+ ACCESS_ONCE(p) = (v); \
+} while (0)
+
+#define smp_load_acquire(p) \
+do { \
+ typeof(p) ___p1 = ACCESS_ONCE(p); \
+ smp_mb(); \
+ return ___p1; \
+} while (0)
+
#endif /* __UNICORE_BARRIER_H__ */
diff --git a/arch/x86/include/asm/barrier.h b/arch/x86/include/asm/barrier.h
index c6cd358a1eec..115ef72b3784 100644
--- a/arch/x86/include/asm/barrier.h
+++ b/arch/x86/include/asm/barrier.h
@@ -100,6 +100,19 @@
#define set_mb(var, value) do { var = value; barrier(); } while (0)
#endif
+#define smp_store_release(p, v) \
+do { \
+ barrier(); \
+ ACCESS_ONCE(p) = (v); \
+} while (0)
+
+#define smp_load_acquire(p) \
+do { \
+ typeof(p) ___p1 = ACCESS_ONCE(p); \
+ barrier(); \
+ return ___p1; \
+} while (0)
+
/*
* Stop RDTSC speculation. This is needed when you need to use RDTSC
* (or get_cycles or vread that possibly accesses the TSC) in a defined
diff --git a/arch/xtensa/include/asm/barrier.h b/arch/xtensa/include/asm/barrier.h
index ef021677d536..e96a674c337a 100644
--- a/arch/xtensa/include/asm/barrier.h
+++ b/arch/xtensa/include/asm/barrier.h
@@ -26,4 +26,17 @@
#define set_mb(var, value) do { var = value; mb(); } while (0)
+#define smp_store_release(p, v) \
+do { \
+ smp_mb(); \
+ ACCESS_ONCE(p) = (v); \
+} while (0)
+
+#define smp_load_acquire(p) \
+do { \
+ typeof(p) ___p1 = ACCESS_ONCE(p); \
+ smp_mb(); \
+ return ___p1; \
+} while (0)
+
#endif /* _XTENSA_SYSTEM_H */
^ permalink raw reply related
* Re: [PATCH v3 1/1] powerpc/embedded6xx: Add support for Motorola/Emerson MVME5100
From: Stephen N Chivers @ 2013-11-04 19:18 UTC (permalink / raw)
To: Geert Uytterhoeven; +Cc: geert.uytterhoeven, linuxppc-dev@lists.ozlabs.org
In-Reply-To: <CAMuHMdUOUf7shC9A+enCt8qQygG=Qw_soeNJnTWQKr90o2gZhA@mail.gmail.com>
geert.uytterhoeven@gmail.com wrote on 11/04/2013 06:59:21 PM:
> From: Geert Uytterhoeven <geert@linux-m68k.org>
> To: Stephen N Chivers/AUS/CSC@CSC
> Cc: "linuxppc-dev@lists.ozlabs.org" <linuxppc-dev@lists.ozlabs.org>
> Date: 11/04/2013 06:59 PM
> Subject: Re: [PATCH v3 1/1] powerpc/embedded6xx: Add support for
> Motorola/Emerson MVME5100
> Sent by: geert.uytterhoeven@gmail.com
>
> On Sun, Nov 3, 2013 at 10:07 PM, Stephen Chivers <schivers@csc.com>
wrote:
> > +++ b/arch/powerpc/boot/dts/mvme5100.dts
> > @@ -0,0 +1,185 @@
> > +/*
> > + * Device Tree Souce for Motorola/Emerson MVME5100.
>
> Source
Ok. Thanks, will be fixed.
>
> (unless this expresses your personal appreciation for device trees ;-)
>
> Gr{oetje,eeting}s,
>
> Geert
>
> --
> Geert Uytterhoeven -- There's lots of Linux beyond ia32 --
> geert@linux-m68k.org
>
> In personal conversations with technical people, I call myself a hacker.
But
> when I'm talking to journalists I just say "programmer" or somethinglike
that.
> -- Linus Torvalds
^ permalink raw reply
* Re: [RFC] arch: Introduce new TSO memory barrier smp_tmb()
From: Paul E. McKenney @ 2013-11-04 20:53 UTC (permalink / raw)
To: Peter Zijlstra
Cc: Michael Neuling, Mathieu Desnoyers, heiko.carstens, Oleg Nesterov,
LKML, Linux PPC dev, Anton Blanchard, Frederic Weisbecker,
Victor Kaplansky, linux, Linus Torvalds, schwidefsky
In-Reply-To: <20131104191127.GW16117@laptop.programming.kicks-ass.net>
On Mon, Nov 04, 2013 at 08:11:27PM +0100, Peter Zijlstra wrote:
> On Mon, Nov 04, 2013 at 08:27:32AM -0800, Paul E. McKenney wrote:
> > All this is leading me to suggest the following shortenings of names:
> >
> > smp_load_with_acquire_semantics() -> smp_load_acquire()
> >
> > smp_store_with_release_semantics() -> smp_store_release()
> >
> > But names aside, the above gets rid of explicit barriers on TSO architectures,
> > allows ARM to avoid full DMB, and allows PowerPC to use lwsync instead of
> > the heavier-weight sync.
>
> A little something like this? Completely guessed at the arm/arm64/ia64
> asm, but at least for those archs I found proper instructions (I hope),
> for x86,sparc,s390 which are TSO we can do with a barrier and PPC like
> said can do with the lwsync, all others fall back to using a smp_mb().
>
> Should probably come with a proper changelog and an addition to _The_
> document.
Maybe something like this for the changelog?
A number of situations currently require the heavyweight smp_mb(),
even though there is no need to order prior stores against later
loads. Many architectures have much cheaper ways to handle these
situations, but the Linux kernel currently has no portable way
to make use of them.
This commit therefore supplies smp_load_acquire() and
smp_store_release() to remedy this situation. The new
smp_load_acquire() primitive orders the specified load against
any subsequent reads or writes, while the new smp_store_release()
primitive orders the specifed store against any prior reads or
writes. These primitives allow array-based circular FIFOs to be
implemented without an smp_mb(), and also allow a theoretical
hole in rcu_assign_pointer() to be closed at no additional
expense on most architectures.
In addition, the RCU experience transitioning from explicit
smp_read_barrier_depends() and smp_wmb() to rcu_dereference()
and rcu_assign_pointer(), respectively resulted in substantial
improvements in readability. It therefore seems likely that
replacing other explicit barriers with smp_load_acquire() and
smp_store_release() will provide similar benefits. It appears
that roughly half of the explicit barriers in core kernel code
might be so replaced.
Some comments below. I believe that opcodes need to be fixed for IA64.
I am unsure of the ifdefs and opcodes for arm64, but the ARM folks should
be able to tell us.
Other than that, for the rest:
Reviewed-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
> ---
> arch/alpha/include/asm/barrier.h | 13 +++++++++++
> arch/arc/include/asm/barrier.h | 13 +++++++++++
> arch/arm/include/asm/barrier.h | 26 +++++++++++++++++++++
> arch/arm64/include/asm/barrier.h | 28 +++++++++++++++++++++++
> arch/avr32/include/asm/barrier.h | 12 ++++++++++
> arch/blackfin/include/asm/barrier.h | 13 +++++++++++
> arch/cris/include/asm/barrier.h | 13 +++++++++++
> arch/frv/include/asm/barrier.h | 13 +++++++++++
> arch/h8300/include/asm/barrier.h | 13 +++++++++++
> arch/hexagon/include/asm/barrier.h | 13 +++++++++++
> arch/ia64/include/asm/barrier.h | 43 +++++++++++++++++++++++++++++++++++
> arch/m32r/include/asm/barrier.h | 13 +++++++++++
> arch/m68k/include/asm/barrier.h | 13 +++++++++++
> arch/metag/include/asm/barrier.h | 13 +++++++++++
> arch/microblaze/include/asm/barrier.h | 13 +++++++++++
> arch/mips/include/asm/barrier.h | 13 +++++++++++
> arch/mn10300/include/asm/barrier.h | 13 +++++++++++
> arch/parisc/include/asm/barrier.h | 13 +++++++++++
> arch/powerpc/include/asm/barrier.h | 15 ++++++++++++
> arch/s390/include/asm/barrier.h | 13 +++++++++++
> arch/score/include/asm/barrier.h | 13 +++++++++++
> arch/sh/include/asm/barrier.h | 13 +++++++++++
> arch/sparc/include/asm/barrier_32.h | 13 +++++++++++
> arch/sparc/include/asm/barrier_64.h | 13 +++++++++++
> arch/tile/include/asm/barrier.h | 13 +++++++++++
> arch/unicore32/include/asm/barrier.h | 13 +++++++++++
> arch/x86/include/asm/barrier.h | 13 +++++++++++
> arch/xtensa/include/asm/barrier.h | 13 +++++++++++
> 28 files changed, 423 insertions(+)
>
> diff --git a/arch/alpha/include/asm/barrier.h b/arch/alpha/include/asm/barrier.h
> index ce8860a0b32d..464139feee97 100644
> --- a/arch/alpha/include/asm/barrier.h
> +++ b/arch/alpha/include/asm/barrier.h
> @@ -29,6 +29,19 @@ __asm__ __volatile__("mb": : :"memory")
> #define smp_read_barrier_depends() do { } while (0)
> #endif
>
> +#define smp_store_release(p, v) \
> +do { \
> + smp_mb(); \
> + ACCESS_ONCE(p) = (v); \
> +} while (0)
> +
> +#define smp_load_acquire(p) \
> +do { \
> + typeof(p) ___p1 = ACCESS_ONCE(p); \
> + smp_mb(); \
> + return ___p1; \
> +} while (0)
> +
Yep, not any alternative to smp_mb() here.
> #define set_mb(var, value) \
> do { var = value; mb(); } while (0)
>
> diff --git a/arch/arc/include/asm/barrier.h b/arch/arc/include/asm/barrier.h
> index f6cb7c4ffb35..a779da846fb5 100644
> --- a/arch/arc/include/asm/barrier.h
> +++ b/arch/arc/include/asm/barrier.h
> @@ -30,6 +30,19 @@
> #define smp_wmb() barrier()
> #endif
>
> +#define smp_store_release(p, v) \
> +do { \
> + smp_mb(); \
> + ACCESS_ONCE(p) = (v); \
> +} while (0)
> +
> +#define smp_load_acquire(p) \
> +do { \
> + typeof(p) ___p1 = ACCESS_ONCE(p); \
> + smp_mb(); \
> + return ___p1; \
> +} while (0)
> +
Appears to be !SMP, so OK.
> #define smp_mb__before_atomic_dec() barrier()
> #define smp_mb__after_atomic_dec() barrier()
> #define smp_mb__before_atomic_inc() barrier()
> diff --git a/arch/arm/include/asm/barrier.h b/arch/arm/include/asm/barrier.h
> index 60f15e274e6d..a804093d6891 100644
> --- a/arch/arm/include/asm/barrier.h
> +++ b/arch/arm/include/asm/barrier.h
> @@ -53,10 +53,36 @@
> #define smp_mb() barrier()
> #define smp_rmb() barrier()
> #define smp_wmb() barrier()
> +
> +#define smp_store_release(p, v) \
> +do { \
> + smp_mb(); \
> + ACCESS_ONCE(p) = (v); \
> +} while (0)
> +
> +#define smp_load_acquire(p, v) \
> +do { \
> + typeof(p) ___p1 = ACCESS_ONCE(p); \
> + smp_mb(); \
> + return ___p1; \
> +} while (0)
> #else
> #define smp_mb() dmb(ish)
> #define smp_rmb() smp_mb()
> #define smp_wmb() dmb(ishst)
> +
Seems like there should be some sort of #ifdef condition to distinguish
between these. My guess is something like:
#if __LINUX_ARM_ARCH__ > 7
But I must defer to the ARM guys. For all I know, they might prefer
arch/arm to stick with smp_mb() and have arch/arm64 do the ldar and stlr.
> +#define smp_store_release(p, v) \
> +do { \
> + asm volatile ("stlr %w0 [%1]" : : "r" (v), "r" (&p) : "memory");\
> +} while (0)
> +
> +#define smp_load_acquire(p) \
> +do { \
> + typeof(p) ___p1; \
> + asm volatile ("ldar %w0, [%1]" \
> + : "=r" (___p1) : "r" (&p) : "memory"); \
> + return ___p1; \
> +} while (0)
> #endif
>
> #define read_barrier_depends() do { } while(0)
> diff --git a/arch/arm64/include/asm/barrier.h b/arch/arm64/include/asm/barrier.h
> index d4a63338a53c..0da2d4ebb9a8 100644
> --- a/arch/arm64/include/asm/barrier.h
> +++ b/arch/arm64/include/asm/barrier.h
> @@ -35,10 +35,38 @@
> #define smp_mb() barrier()
> #define smp_rmb() barrier()
> #define smp_wmb() barrier()
> +
> +#define smp_store_release(p, v) \
> +do { \
> + smp_mb(); \
> + ACCESS_ONCE(p) = (v); \
> +} while (0)
> +
> +#define smp_load_acquire(p, v) \
> +do { \
> + typeof(p) ___p1 = ACCESS_ONCE(p); \
> + smp_mb(); \
> + return ___p1; \
> +} while (0)
> +
> #else
> +
> #define smp_mb() asm volatile("dmb ish" : : : "memory")
> #define smp_rmb() asm volatile("dmb ishld" : : : "memory")
> #define smp_wmb() asm volatile("dmb ishst" : : : "memory")
> +
> +#define smp_store_release(p, v) \
> +do { \
> + asm volatile ("stlr %w0 [%1]" : : "r" (v), "r" (&p) : "memory");\
> +} while (0)
> +
> +#define smp_load_acquire(p) \
> +do { \
> + typeof(p) ___p1; \
> + asm volatile ("ldar %w0, [%1]" \
> + : "=r" (___p1) : "r" (&p) : "memory"); \
> + return ___p1; \
> +} while (0)
> #endif
Ditto on the instruction format. The closest thing I see in the kernel
is "stlr %w1, %0" in arch_write_unlock() and arch_spin_unlock().
>
> #define read_barrier_depends() do { } while(0)
> diff --git a/arch/avr32/include/asm/barrier.h b/arch/avr32/include/asm/barrier.h
> index 0961275373db..a0c48ad684f8 100644
> --- a/arch/avr32/include/asm/barrier.h
> +++ b/arch/avr32/include/asm/barrier.h
> @@ -25,5 +25,17 @@
> # define smp_read_barrier_depends() do { } while(0)
> #endif
>
> +#define smp_store_release(p, v) \
> +do { \
> + smp_mb(); \
> + ACCESS_ONCE(p) = (v); \
> +} while (0)
> +
> +#define smp_load_acquire(p, v) \
> +do { \
> + typeof(p) ___p1 = ACCESS_ONCE(p); \
> + smp_mb(); \
> + return ___p1; \
> +} while (0)
!SMP, so should be OK.
>
> #endif /* __ASM_AVR32_BARRIER_H */
> diff --git a/arch/blackfin/include/asm/barrier.h b/arch/blackfin/include/asm/barrier.h
> index ebb189507dd7..67889d9225d9 100644
> --- a/arch/blackfin/include/asm/barrier.h
> +++ b/arch/blackfin/include/asm/barrier.h
> @@ -45,4 +45,17 @@
> #define set_mb(var, value) do { var = value; mb(); } while (0)
> #define smp_read_barrier_depends() read_barrier_depends()
>
> +#define smp_store_release(p, v) \
> +do { \
> + smp_mb(); \
> + ACCESS_ONCE(p) = (v); \
> +} while (0)
> +
> +#define smp_load_acquire(p, v) \
> +do { \
> + typeof(p) ___p1 = ACCESS_ONCE(p); \
> + smp_mb(); \
> + return ___p1; \
> +} while (0)
> +
Ditto.
> #endif /* _BLACKFIN_BARRIER_H */
> diff --git a/arch/cris/include/asm/barrier.h b/arch/cris/include/asm/barrier.h
> index 198ad7fa6b25..34243dc44ef1 100644
> --- a/arch/cris/include/asm/barrier.h
> +++ b/arch/cris/include/asm/barrier.h
> @@ -22,4 +22,17 @@
> #define smp_read_barrier_depends() do { } while(0)
> #endif
>
> +#define smp_store_release(p, v) \
> +do { \
> + smp_mb(); \
> + ACCESS_ONCE(p) = (v); \
> +} while (0)
> +
> +#define smp_load_acquire(p, v) \
> +do { \
> + typeof(p) ___p1 = ACCESS_ONCE(p); \
> + smp_mb(); \
> + return ___p1; \
> +} while (0)
> +
Ditto.
> #endif /* __ASM_CRIS_BARRIER_H */
> diff --git a/arch/frv/include/asm/barrier.h b/arch/frv/include/asm/barrier.h
> index 06776ad9f5e9..92f89934d4ed 100644
> --- a/arch/frv/include/asm/barrier.h
> +++ b/arch/frv/include/asm/barrier.h
> @@ -26,4 +26,17 @@
> #define set_mb(var, value) \
> do { var = (value); barrier(); } while (0)
>
> +#define smp_store_release(p, v) \
> +do { \
> + smp_mb(); \
> + ACCESS_ONCE(p) = (v); \
> +} while (0)
> +
> +#define smp_load_acquire(p, v) \
> +do { \
> + typeof(p) ___p1 = ACCESS_ONCE(p); \
> + smp_mb(); \
> + return ___p1; \
> +} while (0)
> +
Ditto.
> #endif /* _ASM_BARRIER_H */
> diff --git a/arch/h8300/include/asm/barrier.h b/arch/h8300/include/asm/barrier.h
> index 9e0aa9fc195d..516e9d379e25 100644
> --- a/arch/h8300/include/asm/barrier.h
> +++ b/arch/h8300/include/asm/barrier.h
> @@ -26,4 +26,17 @@
> #define smp_read_barrier_depends() do { } while(0)
> #endif
>
> +#define smp_store_release(p, v) \
> +do { \
> + smp_mb(); \
> + ACCESS_ONCE(p) = (v); \
> +} while (0)
> +
> +#define smp_load_acquire(p, v) \
> +do { \
> + typeof(p) ___p1 = ACCESS_ONCE(p); \
> + smp_mb(); \
> + return ___p1; \
> +} while (0)
> +
And ditto again...
> #endif /* _H8300_BARRIER_H */
> diff --git a/arch/hexagon/include/asm/barrier.h b/arch/hexagon/include/asm/barrier.h
> index 1041a8e70ce8..838a2ebe07a5 100644
> --- a/arch/hexagon/include/asm/barrier.h
> +++ b/arch/hexagon/include/asm/barrier.h
> @@ -38,4 +38,17 @@
> #define set_mb(var, value) \
> do { var = value; mb(); } while (0)
>
> +#define smp_store_release(p, v) \
> +do { \
> + smp_mb(); \
> + ACCESS_ONCE(p) = (v); \
> +} while (0)
> +
> +#define smp_load_acquire(p, v) \
> +do { \
> + typeof(p) ___p1 = ACCESS_ONCE(p); \
> + smp_mb(); \
> + return ___p1; \
> +} while (0)
> +
And again...
> #endif /* _ASM_BARRIER_H */
> diff --git a/arch/ia64/include/asm/barrier.h b/arch/ia64/include/asm/barrier.h
> index 60576e06b6fb..4598d390fabb 100644
> --- a/arch/ia64/include/asm/barrier.h
> +++ b/arch/ia64/include/asm/barrier.h
> @@ -45,11 +45,54 @@
> # define smp_rmb() rmb()
> # define smp_wmb() wmb()
> # define smp_read_barrier_depends() read_barrier_depends()
> +
> +#define smp_store_release(p, v) \
> +do { \
> + switch (sizeof(p)) { \
> + case 4: \
> + asm volatile ("st4.acq [%0]=%1" \
This should be "st4.rel".
> + :: "r" (&p), "r" (v) : "memory"); \
> + break; \
> + case 8: \
> + asm volatile ("st8.acq [%0]=%1" \
And this should be "st8.rel"
> + :: "r" (&p), "r" (v) : "memory"); \
> + break; \
> + } \
> +} while (0)
> +
> +#define smp_load_acquire(p, v) \
> +do { \
> + typeof(p) ___p1; \
> + switch (sizeof(p)) { \
> + case 4: \
> + asm volatile ("ld4.rel %0=[%1]" \
And this should be "ld4.acq".
> + : "=r"(___p1) : "r" (&p) : "memory"); \
> + break; \
> + case 8: \
> + asm volatile ("ld8.rel %0=[%1]" \
And this should be "ld8.acq".
> + : "=r"(___p1) : "r" (&p) : "memory"); \
> + break; \
> + } \
> + return ___p1; \
> +} while (0)
It appears that sizes 2 and 1 are also available, but 4 and 8 seem like
good places to start.
> #else
> # define smp_mb() barrier()
> # define smp_rmb() barrier()
> # define smp_wmb() barrier()
> # define smp_read_barrier_depends() do { } while(0)
> +
> +#define smp_store_release(p, v) \
> +do { \
> + smp_mb(); \
> + ACCESS_ONCE(p) = (v); \
> +} while (0)
> +
> +#define smp_load_acquire(p, v) \
> +do { \
> + typeof(p) ___p1 = ACCESS_ONCE(p); \
> + smp_mb(); \
> + return ___p1; \
> +} while (0)
> #endif
>
> /*
> diff --git a/arch/m32r/include/asm/barrier.h b/arch/m32r/include/asm/barrier.h
> index 6976621efd3f..e5d42bcf90c5 100644
> --- a/arch/m32r/include/asm/barrier.h
> +++ b/arch/m32r/include/asm/barrier.h
> @@ -91,4 +91,17 @@
> #define set_mb(var, value) do { var = value; barrier(); } while (0)
> #endif
>
> +#define smp_store_release(p, v) \
> +do { \
> + smp_mb(); \
> + ACCESS_ONCE(p) = (v); \
> +} while (0)
> +
> +#define smp_load_acquire(p, v) \
> +do { \
> + typeof(p) ___p1 = ACCESS_ONCE(p); \
> + smp_mb(); \
> + return ___p1; \
> +} while (0)
> +
Another !SMP architecture, so looks good.
> #endif /* _ASM_M32R_BARRIER_H */
> diff --git a/arch/m68k/include/asm/barrier.h b/arch/m68k/include/asm/barrier.h
> index 445ce22c23cb..eeb9ecf713cc 100644
> --- a/arch/m68k/include/asm/barrier.h
> +++ b/arch/m68k/include/asm/barrier.h
> @@ -17,4 +17,17 @@
> #define smp_wmb() barrier()
> #define smp_read_barrier_depends() ((void)0)
>
> +#define smp_store_release(p, v) \
> +do { \
> + smp_mb(); \
> + ACCESS_ONCE(p) = (v); \
> +} while (0)
> +
> +#define smp_load_acquire(p, v) \
> +do { \
> + typeof(p) ___p1 = ACCESS_ONCE(p); \
> + smp_mb(); \
> + return ___p1; \
> +} while (0)
> +
Ditto.
> #endif /* _M68K_BARRIER_H */
> diff --git a/arch/metag/include/asm/barrier.h b/arch/metag/include/asm/barrier.h
> index c90bfc6bf648..d8e6f2e4a27c 100644
> --- a/arch/metag/include/asm/barrier.h
> +++ b/arch/metag/include/asm/barrier.h
> @@ -82,4 +82,17 @@ static inline void fence(void)
> #define smp_read_barrier_depends() do { } while (0)
> #define set_mb(var, value) do { var = value; smp_mb(); } while (0)
>
> +#define smp_store_release(p, v) \
> +do { \
> + smp_mb(); \
> + ACCESS_ONCE(p) = (v); \
> +} while (0)
> +
> +#define smp_load_acquire(p, v) \
> +do { \
> + typeof(p) ___p1 = ACCESS_ONCE(p); \
> + smp_mb(); \
> + return ___p1; \
> +} while (0)
> +
This one is a bit unusual, but use of smp_mb() should be safe.
> #endif /* _ASM_METAG_BARRIER_H */
> diff --git a/arch/microblaze/include/asm/barrier.h b/arch/microblaze/include/asm/barrier.h
> index df5be3e87044..a890702061c9 100644
> --- a/arch/microblaze/include/asm/barrier.h
> +++ b/arch/microblaze/include/asm/barrier.h
> @@ -24,4 +24,17 @@
> #define smp_rmb() rmb()
> #define smp_wmb() wmb()
>
> +#define smp_store_release(p, v) \
> +do { \
> + smp_mb(); \
> + ACCESS_ONCE(p) = (v); \
> +} while (0)
> +
> +#define smp_load_acquire(p, v) \
> +do { \
> + typeof(p) ___p1 = ACCESS_ONCE(p); \
> + smp_mb(); \
> + return ___p1; \
> +} while (0)
> +
!SMP only, so good.
> #endif /* _ASM_MICROBLAZE_BARRIER_H */
> diff --git a/arch/mips/include/asm/barrier.h b/arch/mips/include/asm/barrier.h
> index 314ab5532019..e59bcd051f36 100644
> --- a/arch/mips/include/asm/barrier.h
> +++ b/arch/mips/include/asm/barrier.h
> @@ -180,4 +180,17 @@
> #define nudge_writes() mb()
> #endif
>
> +#define smp_store_release(p, v) \
> +do { \
> + smp_mb(); \
> + ACCESS_ONCE(p) = (v); \
> +} while (0)
> +
> +#define smp_load_acquire(p, v) \
> +do { \
> + typeof(p) ___p1 = ACCESS_ONCE(p); \
> + smp_mb(); \
> + return ___p1; \
> +} while (0)
> +
Interesting variety here as well. Again, smp_mb() should be safe.
> #endif /* __ASM_BARRIER_H */
> diff --git a/arch/mn10300/include/asm/barrier.h b/arch/mn10300/include/asm/barrier.h
> index 2bd97a5c8af7..0e6a0608d4a1 100644
> --- a/arch/mn10300/include/asm/barrier.h
> +++ b/arch/mn10300/include/asm/barrier.h
> @@ -34,4 +34,17 @@
> #define read_barrier_depends() do {} while (0)
> #define smp_read_barrier_depends() do {} while (0)
>
> +#define smp_store_release(p, v) \
> +do { \
> + smp_mb(); \
> + ACCESS_ONCE(p) = (v); \
> +} while (0)
> +
> +#define smp_load_acquire(p, v) \
> +do { \
> + typeof(p) ___p1 = ACCESS_ONCE(p); \
> + smp_mb(); \
> + return ___p1; \
> +} while (0)
> +
!SMP, so good.
> #endif /* _ASM_BARRIER_H */
> diff --git a/arch/parisc/include/asm/barrier.h b/arch/parisc/include/asm/barrier.h
> index e77d834aa803..f1145a8594a0 100644
> --- a/arch/parisc/include/asm/barrier.h
> +++ b/arch/parisc/include/asm/barrier.h
> @@ -32,4 +32,17 @@
>
> #define set_mb(var, value) do { var = value; mb(); } while (0)
>
> +#define smp_store_release(p, v) \
> +do { \
> + smp_mb(); \
> + ACCESS_ONCE(p) = (v); \
> +} while (0)
> +
> +#define smp_load_acquire(p, v) \
> +do { \
> + typeof(p) ___p1 = ACCESS_ONCE(p); \
> + smp_mb(); \
> + return ___p1; \
> +} while (0)
> +
Ditto.
> #endif /* __PARISC_BARRIER_H */
> diff --git a/arch/powerpc/include/asm/barrier.h b/arch/powerpc/include/asm/barrier.h
> index ae782254e731..b5cc36791f42 100644
> --- a/arch/powerpc/include/asm/barrier.h
> +++ b/arch/powerpc/include/asm/barrier.h
> @@ -65,4 +65,19 @@
> #define data_barrier(x) \
> asm volatile("twi 0,%0,0; isync" : : "r" (x) : "memory");
>
> +/* use smp_rmb() as that is either lwsync or a barrier() depending on SMP */
> +
> +#define smp_store_release(p, v) \
> +do { \
> + smp_rmb(); \
> + ACCESS_ONCE(p) = (v); \
> +} while (0)
> +
> +#define smp_load_acquire(p, v) \
> +do { \
> + typeof(p) ___p1 = ACCESS_ONCE(p); \
> + smp_rmb(); \
> + return ___p1; \
> +} while (0)
> +
I think that this actually does work, strange though it does look.
> #endif /* _ASM_POWERPC_BARRIER_H */
> diff --git a/arch/s390/include/asm/barrier.h b/arch/s390/include/asm/barrier.h
> index 16760eeb79b0..e8989c40e11c 100644
> --- a/arch/s390/include/asm/barrier.h
> +++ b/arch/s390/include/asm/barrier.h
> @@ -32,4 +32,17 @@
>
> #define set_mb(var, value) do { var = value; mb(); } while (0)
>
> +#define smp_store_release(p, v) \
> +do { \
> + barrier(); \
> + ACCESS_ONCE(p) = (v); \
> +} while (0)
> +
> +#define smp_load_acquire(p, v) \
> +do { \
> + typeof(p) ___p1 = ACCESS_ONCE(p); \
> + barrier(); \
> + return ___p1; \
> +} while (0)
> +
I believe that this is OK as well, but must defer to the s390
maintainers.
> #endif /* __ASM_BARRIER_H */
> diff --git a/arch/score/include/asm/barrier.h b/arch/score/include/asm/barrier.h
> index 0eacb6471e6d..5f101ef8ade9 100644
> --- a/arch/score/include/asm/barrier.h
> +++ b/arch/score/include/asm/barrier.h
> @@ -13,4 +13,17 @@
>
> #define set_mb(var, value) do {var = value; wmb(); } while (0)
>
> +#define smp_store_release(p, v) \
> +do { \
> + smp_mb(); \
> + ACCESS_ONCE(p) = (v); \
> +} while (0)
> +
> +#define smp_load_acquire(p, v) \
> +do { \
> + typeof(p) ___p1 = ACCESS_ONCE(p); \
> + smp_mb(); \
> + return ___p1; \
> +} while (0)
> +
!SMP, so good.
> #endif /* _ASM_SCORE_BARRIER_H */
> diff --git a/arch/sh/include/asm/barrier.h b/arch/sh/include/asm/barrier.h
> index 72c103dae300..611128c2f636 100644
> --- a/arch/sh/include/asm/barrier.h
> +++ b/arch/sh/include/asm/barrier.h
> @@ -51,4 +51,17 @@
>
> #define set_mb(var, value) do { (void)xchg(&var, value); } while (0)
>
> +#define smp_store_release(p, v) \
> +do { \
> + smp_mb(); \
> + ACCESS_ONCE(p) = (v); \
> +} while (0)
> +
> +#define smp_load_acquire(p, v) \
> +do { \
> + typeof(p) ___p1 = ACCESS_ONCE(p); \
> + smp_mb(); \
> + return ___p1; \
> +} while (0)
> +
Use of smp_mb() should be safe here.
> #endif /* __ASM_SH_BARRIER_H */
> diff --git a/arch/sparc/include/asm/barrier_32.h b/arch/sparc/include/asm/barrier_32.h
> index c1b76654ee76..f47f9d51f326 100644
> --- a/arch/sparc/include/asm/barrier_32.h
> +++ b/arch/sparc/include/asm/barrier_32.h
> @@ -12,4 +12,17 @@
> #define smp_wmb() __asm__ __volatile__("":::"memory")
> #define smp_read_barrier_depends() do { } while(0)
>
> +#define smp_store_release(p, v) \
> +do { \
> + smp_mb(); \
> + ACCESS_ONCE(p) = (v); \
> +} while (0)
> +
> +#define smp_load_acquire(p, v) \
> +do { \
> + typeof(p) ___p1 = ACCESS_ONCE(p); \
> + smp_mb(); \
> + return ___p1; \
> +} while (0)
> +
The surrounding code looks to be set up for !SMP. I -thought- that there
were SMP 32-bit SPARC systems, but either way, smp_mb() should be safe.
> #endif /* !(__SPARC_BARRIER_H) */
> diff --git a/arch/sparc/include/asm/barrier_64.h b/arch/sparc/include/asm/barrier_64.h
> index 95d45986f908..77cbe6982ca0 100644
> --- a/arch/sparc/include/asm/barrier_64.h
> +++ b/arch/sparc/include/asm/barrier_64.h
> @@ -53,4 +53,17 @@ do { __asm__ __volatile__("ba,pt %%xcc, 1f\n\t" \
>
> #define smp_read_barrier_depends() do { } while(0)
>
> +#define smp_store_release(p, v) \
> +do { \
> + barrier(); \
> + ACCESS_ONCE(p) = (v); \
> +} while (0)
> +
> +#define smp_load_acquire(p, v) \
> +do { \
> + typeof(p) ___p1 = ACCESS_ONCE(p); \
> + barrier(); \
> + return ___p1; \
> +} while (0)
> +
SPARC64 is TSO, so looks good.
> #endif /* !(__SPARC64_BARRIER_H) */
> diff --git a/arch/tile/include/asm/barrier.h b/arch/tile/include/asm/barrier.h
> index a9a73da5865d..4d5330d4fd31 100644
> --- a/arch/tile/include/asm/barrier.h
> +++ b/arch/tile/include/asm/barrier.h
> @@ -140,5 +140,18 @@ mb_incoherent(void)
> #define set_mb(var, value) \
> do { var = value; mb(); } while (0)
>
> +#define smp_store_release(p, v) \
> +do { \
> + smp_mb(); \
> + ACCESS_ONCE(p) = (v); \
> +} while (0)
> +
> +#define smp_load_acquire(p, v) \
> +do { \
> + typeof(p) ___p1 = ACCESS_ONCE(p); \
> + smp_mb(); \
> + return ___p1; \
> +} while (0)
> +
The __mb_incoherent() in the surrounding code looks scary, but smp_mb()
should suffice here as well as elsewhere.
> #endif /* !__ASSEMBLY__ */
> #endif /* _ASM_TILE_BARRIER_H */
> diff --git a/arch/unicore32/include/asm/barrier.h b/arch/unicore32/include/asm/barrier.h
> index a6620e5336b6..5471ff6aae10 100644
> --- a/arch/unicore32/include/asm/barrier.h
> +++ b/arch/unicore32/include/asm/barrier.h
> @@ -25,4 +25,17 @@
>
> #define set_mb(var, value) do { var = value; smp_mb(); } while (0)
>
> +#define smp_store_release(p, v) \
> +do { \
> + smp_mb(); \
> + ACCESS_ONCE(p) = (v); \
> +} while (0)
> +
> +#define smp_load_acquire(p, v) \
> +do { \
> + typeof(p) ___p1 = ACCESS_ONCE(p); \
> + smp_mb(); \
> + return ___p1; \
> +} while (0)
> +
!SMP, so good.
> #endif /* __UNICORE_BARRIER_H__ */
> diff --git a/arch/x86/include/asm/barrier.h b/arch/x86/include/asm/barrier.h
> index c6cd358a1eec..a7fd8201ab09 100644
> --- a/arch/x86/include/asm/barrier.h
> +++ b/arch/x86/include/asm/barrier.h
> @@ -100,6 +100,19 @@
> #define set_mb(var, value) do { var = value; barrier(); } while (0)
> #endif
>
> +#define smp_store_release(p, v) \
> +do { \
> + barrier(); \
> + ACCESS_ONCE(p) = (v); \
> +} while (0)
> +
> +#define smp_load_acquire(p, v) \
> +do { \
> + typeof(p) ___p1 = ACCESS_ONCE(p); \
> + barrier(); \
> + return ___p1; \
> +} while (0)
> +
TSO, so good.
> /*
> * Stop RDTSC speculation. This is needed when you need to use RDTSC
> * (or get_cycles or vread that possibly accesses the TSC) in a defined
> diff --git a/arch/xtensa/include/asm/barrier.h b/arch/xtensa/include/asm/barrier.h
> index ef021677d536..703d511add49 100644
> --- a/arch/xtensa/include/asm/barrier.h
> +++ b/arch/xtensa/include/asm/barrier.h
> @@ -26,4 +26,17 @@
>
> #define set_mb(var, value) do { var = value; mb(); } while (0)
>
> +#define smp_store_release(p, v) \
> +do { \
> + smp_mb(); \
> + ACCESS_ONCE(p) = (v); \
> +} while (0)
> +
> +#define smp_load_acquire(p, v) \
> +do { \
> + typeof(p) ___p1 = ACCESS_ONCE(p); \
> + smp_mb(); \
> + return ___p1; \
> +} while (0)
> +
The use of smp_mb() should be safe, so good. Looks like xtensa orders
reads, but not writes -- interesting...
> #endif /* _XTENSA_SYSTEM_H */
>
^ permalink raw reply
* Re: [RFC] arch: Introduce new TSO memory barrier smp_tmb()
From: Paul E. McKenney @ 2013-11-04 20:54 UTC (permalink / raw)
To: Peter Zijlstra
Cc: Michael Neuling, Mathieu Desnoyers, Oleg Nesterov, LKML,
Linux PPC dev, Anton Blanchard, Frederic Weisbecker,
Victor Kaplansky, Linus Torvalds
In-Reply-To: <20131104191811.GS2490@laptop.programming.kicks-ass.net>
On Mon, Nov 04, 2013 at 08:18:11PM +0100, Peter Zijlstra wrote:
> On Mon, Nov 04, 2013 at 08:11:27PM +0100, Peter Zijlstra wrote:
> > +#define smp_load_acquire(p, v) \
>
> I R idiot!! :-)
OK, I did miss this one as well... :-/
Thanx, Paul
> ---
> arch/alpha/include/asm/barrier.h | 13 +++++++++++
> arch/arc/include/asm/barrier.h | 13 +++++++++++
> arch/arm/include/asm/barrier.h | 26 +++++++++++++++++++++
> arch/arm64/include/asm/barrier.h | 28 +++++++++++++++++++++++
> arch/avr32/include/asm/barrier.h | 12 ++++++++++
> arch/blackfin/include/asm/barrier.h | 13 +++++++++++
> arch/cris/include/asm/barrier.h | 13 +++++++++++
> arch/frv/include/asm/barrier.h | 13 +++++++++++
> arch/h8300/include/asm/barrier.h | 13 +++++++++++
> arch/hexagon/include/asm/barrier.h | 13 +++++++++++
> arch/ia64/include/asm/barrier.h | 43 +++++++++++++++++++++++++++++++++++
> arch/m32r/include/asm/barrier.h | 13 +++++++++++
> arch/m68k/include/asm/barrier.h | 13 +++++++++++
> arch/metag/include/asm/barrier.h | 13 +++++++++++
> arch/microblaze/include/asm/barrier.h | 13 +++++++++++
> arch/mips/include/asm/barrier.h | 13 +++++++++++
> arch/mn10300/include/asm/barrier.h | 13 +++++++++++
> arch/parisc/include/asm/barrier.h | 13 +++++++++++
> arch/powerpc/include/asm/barrier.h | 15 ++++++++++++
> arch/s390/include/asm/barrier.h | 13 +++++++++++
> arch/score/include/asm/barrier.h | 13 +++++++++++
> arch/sh/include/asm/barrier.h | 13 +++++++++++
> arch/sparc/include/asm/barrier_32.h | 13 +++++++++++
> arch/sparc/include/asm/barrier_64.h | 13 +++++++++++
> arch/tile/include/asm/barrier.h | 13 +++++++++++
> arch/unicore32/include/asm/barrier.h | 13 +++++++++++
> arch/x86/include/asm/barrier.h | 13 +++++++++++
> arch/xtensa/include/asm/barrier.h | 13 +++++++++++
> 28 files changed, 423 insertions(+)
>
> diff --git a/arch/alpha/include/asm/barrier.h b/arch/alpha/include/asm/barrier.h
> index ce8860a0b32d..464139feee97 100644
> --- a/arch/alpha/include/asm/barrier.h
> +++ b/arch/alpha/include/asm/barrier.h
> @@ -29,6 +29,19 @@ __asm__ __volatile__("mb": : :"memory")
> #define smp_read_barrier_depends() do { } while (0)
> #endif
>
> +#define smp_store_release(p, v) \
> +do { \
> + smp_mb(); \
> + ACCESS_ONCE(p) = (v); \
> +} while (0)
> +
> +#define smp_load_acquire(p) \
> +do { \
> + typeof(p) ___p1 = ACCESS_ONCE(p); \
> + smp_mb(); \
> + return ___p1; \
> +} while (0)
> +
> #define set_mb(var, value) \
> do { var = value; mb(); } while (0)
>
> diff --git a/arch/arc/include/asm/barrier.h b/arch/arc/include/asm/barrier.h
> index f6cb7c4ffb35..a779da846fb5 100644
> --- a/arch/arc/include/asm/barrier.h
> +++ b/arch/arc/include/asm/barrier.h
> @@ -30,6 +30,19 @@
> #define smp_wmb() barrier()
> #endif
>
> +#define smp_store_release(p, v) \
> +do { \
> + smp_mb(); \
> + ACCESS_ONCE(p) = (v); \
> +} while (0)
> +
> +#define smp_load_acquire(p) \
> +do { \
> + typeof(p) ___p1 = ACCESS_ONCE(p); \
> + smp_mb(); \
> + return ___p1; \
> +} while (0)
> +
> #define smp_mb__before_atomic_dec() barrier()
> #define smp_mb__after_atomic_dec() barrier()
> #define smp_mb__before_atomic_inc() barrier()
> diff --git a/arch/arm/include/asm/barrier.h b/arch/arm/include/asm/barrier.h
> index 60f15e274e6d..4ada4720bdeb 100644
> --- a/arch/arm/include/asm/barrier.h
> +++ b/arch/arm/include/asm/barrier.h
> @@ -53,10 +53,36 @@
> #define smp_mb() barrier()
> #define smp_rmb() barrier()
> #define smp_wmb() barrier()
> +
> +#define smp_store_release(p, v) \
> +do { \
> + smp_mb(); \
> + ACCESS_ONCE(p) = (v); \
> +} while (0)
> +
> +#define smp_load_acquire(p) \
> +do { \
> + typeof(p) ___p1 = ACCESS_ONCE(p); \
> + smp_mb(); \
> + return ___p1; \
> +} while (0)
> #else
> #define smp_mb() dmb(ish)
> #define smp_rmb() smp_mb()
> #define smp_wmb() dmb(ishst)
> +
> +#define smp_store_release(p, v) \
> +do { \
> + asm volatile ("stlr %w0 [%1]" : : "r" (v), "r" (&p) : "memory");\
> +} while (0)
> +
> +#define smp_load_acquire(p) \
> +do { \
> + typeof(p) ___p1; \
> + asm volatile ("ldar %w0, [%1]" \
> + : "=r" (___p1) : "r" (&p) : "memory"); \
> + return ___p1; \
> +} while (0)
> #endif
>
> #define read_barrier_depends() do { } while(0)
> diff --git a/arch/arm64/include/asm/barrier.h b/arch/arm64/include/asm/barrier.h
> index d4a63338a53c..3dfddc0416f6 100644
> --- a/arch/arm64/include/asm/barrier.h
> +++ b/arch/arm64/include/asm/barrier.h
> @@ -35,10 +35,38 @@
> #define smp_mb() barrier()
> #define smp_rmb() barrier()
> #define smp_wmb() barrier()
> +
> +#define smp_store_release(p, v) \
> +do { \
> + smp_mb(); \
> + ACCESS_ONCE(p) = (v); \
> +} while (0)
> +
> +#define smp_load_acquire(p) \
> +do { \
> + typeof(p) ___p1 = ACCESS_ONCE(p); \
> + smp_mb(); \
> + return ___p1; \
> +} while (0)
> +
> #else
> +
> #define smp_mb() asm volatile("dmb ish" : : : "memory")
> #define smp_rmb() asm volatile("dmb ishld" : : : "memory")
> #define smp_wmb() asm volatile("dmb ishst" : : : "memory")
> +
> +#define smp_store_release(p, v) \
> +do { \
> + asm volatile ("stlr %w0 [%1]" : : "r" (v), "r" (&p) : "memory");\
> +} while (0)
> +
> +#define smp_load_acquire(p) \
> +do { \
> + typeof(p) ___p1; \
> + asm volatile ("ldar %w0, [%1]" \
> + : "=r" (___p1) : "r" (&p) : "memory"); \
> + return ___p1; \
> +} while (0)
> #endif
>
> #define read_barrier_depends() do { } while(0)
> diff --git a/arch/avr32/include/asm/barrier.h b/arch/avr32/include/asm/barrier.h
> index 0961275373db..8fd164648e71 100644
> --- a/arch/avr32/include/asm/barrier.h
> +++ b/arch/avr32/include/asm/barrier.h
> @@ -25,5 +25,17 @@
> # define smp_read_barrier_depends() do { } while(0)
> #endif
>
> +#define smp_store_release(p, v) \
> +do { \
> + smp_mb(); \
> + ACCESS_ONCE(p) = (v); \
> +} while (0)
> +
> +#define smp_load_acquire(p) \
> +do { \
> + typeof(p) ___p1 = ACCESS_ONCE(p); \
> + smp_mb(); \
> + return ___p1; \
> +} while (0)
>
> #endif /* __ASM_AVR32_BARRIER_H */
> diff --git a/arch/blackfin/include/asm/barrier.h b/arch/blackfin/include/asm/barrier.h
> index ebb189507dd7..c8b85bba843f 100644
> --- a/arch/blackfin/include/asm/barrier.h
> +++ b/arch/blackfin/include/asm/barrier.h
> @@ -45,4 +45,17 @@
> #define set_mb(var, value) do { var = value; mb(); } while (0)
> #define smp_read_barrier_depends() read_barrier_depends()
>
> +#define smp_store_release(p, v) \
> +do { \
> + smp_mb(); \
> + ACCESS_ONCE(p) = (v); \
> +} while (0)
> +
> +#define smp_load_acquire(p) \
> +do { \
> + typeof(p) ___p1 = ACCESS_ONCE(p); \
> + smp_mb(); \
> + return ___p1; \
> +} while (0)
> +
> #endif /* _BLACKFIN_BARRIER_H */
> diff --git a/arch/cris/include/asm/barrier.h b/arch/cris/include/asm/barrier.h
> index 198ad7fa6b25..26f21f5d1d15 100644
> --- a/arch/cris/include/asm/barrier.h
> +++ b/arch/cris/include/asm/barrier.h
> @@ -22,4 +22,17 @@
> #define smp_read_barrier_depends() do { } while(0)
> #endif
>
> +#define smp_store_release(p, v) \
> +do { \
> + smp_mb(); \
> + ACCESS_ONCE(p) = (v); \
> +} while (0)
> +
> +#define smp_load_acquire(p) \
> +do { \
> + typeof(p) ___p1 = ACCESS_ONCE(p); \
> + smp_mb(); \
> + return ___p1; \
> +} while (0)
> +
> #endif /* __ASM_CRIS_BARRIER_H */
> diff --git a/arch/frv/include/asm/barrier.h b/arch/frv/include/asm/barrier.h
> index 06776ad9f5e9..4569028382fa 100644
> --- a/arch/frv/include/asm/barrier.h
> +++ b/arch/frv/include/asm/barrier.h
> @@ -26,4 +26,17 @@
> #define set_mb(var, value) \
> do { var = (value); barrier(); } while (0)
>
> +#define smp_store_release(p, v) \
> +do { \
> + smp_mb(); \
> + ACCESS_ONCE(p) = (v); \
> +} while (0)
> +
> +#define smp_load_acquire(p) \
> +do { \
> + typeof(p) ___p1 = ACCESS_ONCE(p); \
> + smp_mb(); \
> + return ___p1; \
> +} while (0)
> +
> #endif /* _ASM_BARRIER_H */
> diff --git a/arch/h8300/include/asm/barrier.h b/arch/h8300/include/asm/barrier.h
> index 9e0aa9fc195d..45d36738814d 100644
> --- a/arch/h8300/include/asm/barrier.h
> +++ b/arch/h8300/include/asm/barrier.h
> @@ -26,4 +26,17 @@
> #define smp_read_barrier_depends() do { } while(0)
> #endif
>
> +#define smp_store_release(p, v) \
> +do { \
> + smp_mb(); \
> + ACCESS_ONCE(p) = (v); \
> +} while (0)
> +
> +#define smp_load_acquire(p) \
> +do { \
> + typeof(p) ___p1 = ACCESS_ONCE(p); \
> + smp_mb(); \
> + return ___p1; \
> +} while (0)
> +
> #endif /* _H8300_BARRIER_H */
> diff --git a/arch/hexagon/include/asm/barrier.h b/arch/hexagon/include/asm/barrier.h
> index 1041a8e70ce8..d88d54bd2e6e 100644
> --- a/arch/hexagon/include/asm/barrier.h
> +++ b/arch/hexagon/include/asm/barrier.h
> @@ -38,4 +38,17 @@
> #define set_mb(var, value) \
> do { var = value; mb(); } while (0)
>
> +#define smp_store_release(p, v) \
> +do { \
> + smp_mb(); \
> + ACCESS_ONCE(p) = (v); \
> +} while (0)
> +
> +#define smp_load_acquire(p) \
> +do { \
> + typeof(p) ___p1 = ACCESS_ONCE(p); \
> + smp_mb(); \
> + return ___p1; \
> +} while (0)
> +
> #endif /* _ASM_BARRIER_H */
> diff --git a/arch/ia64/include/asm/barrier.h b/arch/ia64/include/asm/barrier.h
> index 60576e06b6fb..b7f1a8aa03af 100644
> --- a/arch/ia64/include/asm/barrier.h
> +++ b/arch/ia64/include/asm/barrier.h
> @@ -45,11 +45,54 @@
> # define smp_rmb() rmb()
> # define smp_wmb() wmb()
> # define smp_read_barrier_depends() read_barrier_depends()
> +
> +#define smp_store_release(p, v) \
> +do { \
> + switch (sizeof(p)) { \
> + case 4: \
> + asm volatile ("st4.acq [%0]=%1" \
> + :: "r" (&p), "r" (v) : "memory"); \
> + break; \
> + case 8: \
> + asm volatile ("st8.acq [%0]=%1" \
> + :: "r" (&p), "r" (v) : "memory"); \
> + break; \
> + } \
> +} while (0)
> +
> +#define smp_load_acquire(p) \
> +do { \
> + typeof(p) ___p1; \
> + switch (sizeof(p)) { \
> + case 4: \
> + asm volatile ("ld4.rel %0=[%1]" \
> + : "=r"(___p1) : "r" (&p) : "memory"); \
> + break; \
> + case 8: \
> + asm volatile ("ld8.rel %0=[%1]" \
> + : "=r"(___p1) : "r" (&p) : "memory"); \
> + break; \
> + } \
> + return ___p1; \
> +} while (0)
> #else
> # define smp_mb() barrier()
> # define smp_rmb() barrier()
> # define smp_wmb() barrier()
> # define smp_read_barrier_depends() do { } while(0)
> +
> +#define smp_store_release(p, v) \
> +do { \
> + smp_mb(); \
> + ACCESS_ONCE(p) = (v); \
> +} while (0)
> +
> +#define smp_load_acquire(p) \
> +do { \
> + typeof(p) ___p1 = ACCESS_ONCE(p); \
> + smp_mb(); \
> + return ___p1; \
> +} while (0)
> #endif
>
> /*
> diff --git a/arch/m32r/include/asm/barrier.h b/arch/m32r/include/asm/barrier.h
> index 6976621efd3f..d78612289cb2 100644
> --- a/arch/m32r/include/asm/barrier.h
> +++ b/arch/m32r/include/asm/barrier.h
> @@ -91,4 +91,17 @@
> #define set_mb(var, value) do { var = value; barrier(); } while (0)
> #endif
>
> +#define smp_store_release(p, v) \
> +do { \
> + smp_mb(); \
> + ACCESS_ONCE(p) = (v); \
> +} while (0)
> +
> +#define smp_load_acquire(p) \
> +do { \
> + typeof(p) ___p1 = ACCESS_ONCE(p); \
> + smp_mb(); \
> + return ___p1; \
> +} while (0)
> +
> #endif /* _ASM_M32R_BARRIER_H */
> diff --git a/arch/m68k/include/asm/barrier.h b/arch/m68k/include/asm/barrier.h
> index 445ce22c23cb..1e63b11c424c 100644
> --- a/arch/m68k/include/asm/barrier.h
> +++ b/arch/m68k/include/asm/barrier.h
> @@ -17,4 +17,17 @@
> #define smp_wmb() barrier()
> #define smp_read_barrier_depends() ((void)0)
>
> +#define smp_store_release(p, v) \
> +do { \
> + smp_mb(); \
> + ACCESS_ONCE(p) = (v); \
> +} while (0)
> +
> +#define smp_load_acquire(p) \
> +do { \
> + typeof(p) ___p1 = ACCESS_ONCE(p); \
> + smp_mb(); \
> + return ___p1; \
> +} while (0)
> +
> #endif /* _M68K_BARRIER_H */
> diff --git a/arch/metag/include/asm/barrier.h b/arch/metag/include/asm/barrier.h
> index c90bfc6bf648..9ffd0b167f07 100644
> --- a/arch/metag/include/asm/barrier.h
> +++ b/arch/metag/include/asm/barrier.h
> @@ -82,4 +82,17 @@ static inline void fence(void)
> #define smp_read_barrier_depends() do { } while (0)
> #define set_mb(var, value) do { var = value; smp_mb(); } while (0)
>
> +#define smp_store_release(p, v) \
> +do { \
> + smp_mb(); \
> + ACCESS_ONCE(p) = (v); \
> +} while (0)
> +
> +#define smp_load_acquire(p) \
> +do { \
> + typeof(p) ___p1 = ACCESS_ONCE(p); \
> + smp_mb(); \
> + return ___p1; \
> +} while (0)
> +
> #endif /* _ASM_METAG_BARRIER_H */
> diff --git a/arch/microblaze/include/asm/barrier.h b/arch/microblaze/include/asm/barrier.h
> index df5be3e87044..db0b5e205ce3 100644
> --- a/arch/microblaze/include/asm/barrier.h
> +++ b/arch/microblaze/include/asm/barrier.h
> @@ -24,4 +24,17 @@
> #define smp_rmb() rmb()
> #define smp_wmb() wmb()
>
> +#define smp_store_release(p, v) \
> +do { \
> + smp_mb(); \
> + ACCESS_ONCE(p) = (v); \
> +} while (0)
> +
> +#define smp_load_acquire(p) \
> +do { \
> + typeof(p) ___p1 = ACCESS_ONCE(p); \
> + smp_mb(); \
> + return ___p1; \
> +} while (0)
> +
> #endif /* _ASM_MICROBLAZE_BARRIER_H */
> diff --git a/arch/mips/include/asm/barrier.h b/arch/mips/include/asm/barrier.h
> index 314ab5532019..8031afcc7f64 100644
> --- a/arch/mips/include/asm/barrier.h
> +++ b/arch/mips/include/asm/barrier.h
> @@ -180,4 +180,17 @@
> #define nudge_writes() mb()
> #endif
>
> +#define smp_store_release(p, v) \
> +do { \
> + smp_mb(); \
> + ACCESS_ONCE(p) = (v); \
> +} while (0)
> +
> +#define smp_load_acquire(p) \
> +do { \
> + typeof(p) ___p1 = ACCESS_ONCE(p); \
> + smp_mb(); \
> + return ___p1; \
> +} while (0)
> +
> #endif /* __ASM_BARRIER_H */
> diff --git a/arch/mn10300/include/asm/barrier.h b/arch/mn10300/include/asm/barrier.h
> index 2bd97a5c8af7..e822ff76f498 100644
> --- a/arch/mn10300/include/asm/barrier.h
> +++ b/arch/mn10300/include/asm/barrier.h
> @@ -34,4 +34,17 @@
> #define read_barrier_depends() do {} while (0)
> #define smp_read_barrier_depends() do {} while (0)
>
> +#define smp_store_release(p, v) \
> +do { \
> + smp_mb(); \
> + ACCESS_ONCE(p) = (v); \
> +} while (0)
> +
> +#define smp_load_acquire(p) \
> +do { \
> + typeof(p) ___p1 = ACCESS_ONCE(p); \
> + smp_mb(); \
> + return ___p1; \
> +} while (0)
> +
> #endif /* _ASM_BARRIER_H */
> diff --git a/arch/parisc/include/asm/barrier.h b/arch/parisc/include/asm/barrier.h
> index e77d834aa803..58757747f873 100644
> --- a/arch/parisc/include/asm/barrier.h
> +++ b/arch/parisc/include/asm/barrier.h
> @@ -32,4 +32,17 @@
>
> #define set_mb(var, value) do { var = value; mb(); } while (0)
>
> +#define smp_store_release(p, v) \
> +do { \
> + smp_mb(); \
> + ACCESS_ONCE(p) = (v); \
> +} while (0)
> +
> +#define smp_load_acquire(p) \
> +do { \
> + typeof(p) ___p1 = ACCESS_ONCE(p); \
> + smp_mb(); \
> + return ___p1; \
> +} while (0)
> +
> #endif /* __PARISC_BARRIER_H */
> diff --git a/arch/powerpc/include/asm/barrier.h b/arch/powerpc/include/asm/barrier.h
> index ae782254e731..54922626b356 100644
> --- a/arch/powerpc/include/asm/barrier.h
> +++ b/arch/powerpc/include/asm/barrier.h
> @@ -65,4 +65,19 @@
> #define data_barrier(x) \
> asm volatile("twi 0,%0,0; isync" : : "r" (x) : "memory");
>
> +/* use smp_rmb() as that is either lwsync or a barrier() depending on SMP */
> +
> +#define smp_store_release(p, v) \
> +do { \
> + smp_rmb(); \
> + ACCESS_ONCE(p) = (v); \
> +} while (0)
> +
> +#define smp_load_acquire(p) \
> +do { \
> + typeof(p) ___p1 = ACCESS_ONCE(p); \
> + smp_rmb(); \
> + return ___p1; \
> +} while (0)
> +
> #endif /* _ASM_POWERPC_BARRIER_H */
> diff --git a/arch/s390/include/asm/barrier.h b/arch/s390/include/asm/barrier.h
> index 16760eeb79b0..babf928649a4 100644
> --- a/arch/s390/include/asm/barrier.h
> +++ b/arch/s390/include/asm/barrier.h
> @@ -32,4 +32,17 @@
>
> #define set_mb(var, value) do { var = value; mb(); } while (0)
>
> +#define smp_store_release(p, v) \
> +do { \
> + barrier(); \
> + ACCESS_ONCE(p) = (v); \
> +} while (0)
> +
> +#define smp_load_acquire(p) \
> +do { \
> + typeof(p) ___p1 = ACCESS_ONCE(p); \
> + barrier(); \
> + return ___p1; \
> +} while (0)
> +
> #endif /* __ASM_BARRIER_H */
> diff --git a/arch/score/include/asm/barrier.h b/arch/score/include/asm/barrier.h
> index 0eacb6471e6d..5905ea57a104 100644
> --- a/arch/score/include/asm/barrier.h
> +++ b/arch/score/include/asm/barrier.h
> @@ -13,4 +13,17 @@
>
> #define set_mb(var, value) do {var = value; wmb(); } while (0)
>
> +#define smp_store_release(p, v) \
> +do { \
> + smp_mb(); \
> + ACCESS_ONCE(p) = (v); \
> +} while (0)
> +
> +#define smp_load_acquire(p) \
> +do { \
> + typeof(p) ___p1 = ACCESS_ONCE(p); \
> + smp_mb(); \
> + return ___p1; \
> +} while (0)
> +
> #endif /* _ASM_SCORE_BARRIER_H */
> diff --git a/arch/sh/include/asm/barrier.h b/arch/sh/include/asm/barrier.h
> index 72c103dae300..379f500023b6 100644
> --- a/arch/sh/include/asm/barrier.h
> +++ b/arch/sh/include/asm/barrier.h
> @@ -51,4 +51,17 @@
>
> #define set_mb(var, value) do { (void)xchg(&var, value); } while (0)
>
> +#define smp_store_release(p, v) \
> +do { \
> + smp_mb(); \
> + ACCESS_ONCE(p) = (v); \
> +} while (0)
> +
> +#define smp_load_acquire(p) \
> +do { \
> + typeof(p) ___p1 = ACCESS_ONCE(p); \
> + smp_mb(); \
> + return ___p1; \
> +} while (0)
> +
> #endif /* __ASM_SH_BARRIER_H */
> diff --git a/arch/sparc/include/asm/barrier_32.h b/arch/sparc/include/asm/barrier_32.h
> index c1b76654ee76..1649081d1b86 100644
> --- a/arch/sparc/include/asm/barrier_32.h
> +++ b/arch/sparc/include/asm/barrier_32.h
> @@ -12,4 +12,17 @@
> #define smp_wmb() __asm__ __volatile__("":::"memory")
> #define smp_read_barrier_depends() do { } while(0)
>
> +#define smp_store_release(p, v) \
> +do { \
> + smp_mb(); \
> + ACCESS_ONCE(p) = (v); \
> +} while (0)
> +
> +#define smp_load_acquire(p) \
> +do { \
> + typeof(p) ___p1 = ACCESS_ONCE(p); \
> + smp_mb(); \
> + return ___p1; \
> +} while (0)
> +
> #endif /* !(__SPARC_BARRIER_H) */
> diff --git a/arch/sparc/include/asm/barrier_64.h b/arch/sparc/include/asm/barrier_64.h
> index 95d45986f908..5e23ced0a29a 100644
> --- a/arch/sparc/include/asm/barrier_64.h
> +++ b/arch/sparc/include/asm/barrier_64.h
> @@ -53,4 +53,17 @@ do { __asm__ __volatile__("ba,pt %%xcc, 1f\n\t" \
>
> #define smp_read_barrier_depends() do { } while(0)
>
> +#define smp_store_release(p, v) \
> +do { \
> + barrier(); \
> + ACCESS_ONCE(p) = (v); \
> +} while (0)
> +
> +#define smp_load_acquire(p) \
> +do { \
> + typeof(p) ___p1 = ACCESS_ONCE(p); \
> + barrier(); \
> + return ___p1; \
> +} while (0)
> +
> #endif /* !(__SPARC64_BARRIER_H) */
> diff --git a/arch/tile/include/asm/barrier.h b/arch/tile/include/asm/barrier.h
> index a9a73da5865d..1f08318db3c0 100644
> --- a/arch/tile/include/asm/barrier.h
> +++ b/arch/tile/include/asm/barrier.h
> @@ -140,5 +140,18 @@ mb_incoherent(void)
> #define set_mb(var, value) \
> do { var = value; mb(); } while (0)
>
> +#define smp_store_release(p, v) \
> +do { \
> + smp_mb(); \
> + ACCESS_ONCE(p) = (v); \
> +} while (0)
> +
> +#define smp_load_acquire(p) \
> +do { \
> + typeof(p) ___p1 = ACCESS_ONCE(p); \
> + smp_mb(); \
> + return ___p1; \
> +} while (0)
> +
> #endif /* !__ASSEMBLY__ */
> #endif /* _ASM_TILE_BARRIER_H */
> diff --git a/arch/unicore32/include/asm/barrier.h b/arch/unicore32/include/asm/barrier.h
> index a6620e5336b6..fa8bf69d9a09 100644
> --- a/arch/unicore32/include/asm/barrier.h
> +++ b/arch/unicore32/include/asm/barrier.h
> @@ -25,4 +25,17 @@
>
> #define set_mb(var, value) do { var = value; smp_mb(); } while (0)
>
> +#define smp_store_release(p, v) \
> +do { \
> + smp_mb(); \
> + ACCESS_ONCE(p) = (v); \
> +} while (0)
> +
> +#define smp_load_acquire(p) \
> +do { \
> + typeof(p) ___p1 = ACCESS_ONCE(p); \
> + smp_mb(); \
> + return ___p1; \
> +} while (0)
> +
> #endif /* __UNICORE_BARRIER_H__ */
> diff --git a/arch/x86/include/asm/barrier.h b/arch/x86/include/asm/barrier.h
> index c6cd358a1eec..115ef72b3784 100644
> --- a/arch/x86/include/asm/barrier.h
> +++ b/arch/x86/include/asm/barrier.h
> @@ -100,6 +100,19 @@
> #define set_mb(var, value) do { var = value; barrier(); } while (0)
> #endif
>
> +#define smp_store_release(p, v) \
> +do { \
> + barrier(); \
> + ACCESS_ONCE(p) = (v); \
> +} while (0)
> +
> +#define smp_load_acquire(p) \
> +do { \
> + typeof(p) ___p1 = ACCESS_ONCE(p); \
> + barrier(); \
> + return ___p1; \
> +} while (0)
> +
> /*
> * Stop RDTSC speculation. This is needed when you need to use RDTSC
> * (or get_cycles or vread that possibly accesses the TSC) in a defined
> diff --git a/arch/xtensa/include/asm/barrier.h b/arch/xtensa/include/asm/barrier.h
> index ef021677d536..e96a674c337a 100644
> --- a/arch/xtensa/include/asm/barrier.h
> +++ b/arch/xtensa/include/asm/barrier.h
> @@ -26,4 +26,17 @@
>
> #define set_mb(var, value) do { var = value; mb(); } while (0)
>
> +#define smp_store_release(p, v) \
> +do { \
> + smp_mb(); \
> + ACCESS_ONCE(p) = (v); \
> +} while (0)
> +
> +#define smp_load_acquire(p) \
> +do { \
> + typeof(p) ___p1 = ACCESS_ONCE(p); \
> + smp_mb(); \
> + return ___p1; \
> +} while (0)
> +
> #endif /* _XTENSA_SYSTEM_H */
>
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox