sparc64 cmpxchg is not a full memory barrier anymore ?

sparc64 cmpxchg is not a full memory barrier anymore ?

[Mathieu Desnoyers]

Hi David,

I took a look at the current sparc64 cmpxchg implementation in the Linux
kernel and stumbled on this commit:

commit 293666b7a17cb7a389fc274980439212386a19c4
Author: David S. Miller <davem@davemloft.net>
Date:   Sat Nov 15 13:33:25 2008 -0800

    sparc64: Stop using memory barriers for atomics and locks.
    
    The kernel always executes in the TSO memory model now,
    so none of this stuff is necessary any more.
    
    With helpful feedback from Nick Piggin.
    
    Signed-off-by: David S. Miller <davem@davemloft.net>

Reading p. 152 A.9 Compare and Swap, I see that the cas instruction does
not imply a memory barrier.

Reading http://docs.sun.com/app/docs/doc/801-6678/6i11oelck?a=view

I see that:

"TSO guarantees that the store, FLUSH, and atomic load-store
instructions of all processors appear to be executed by memory serially
in a single order called the memory order. Furthermore, the sequence of
store, FLUSH, and atomic load-store instructions in the memory order for
a given processor is identical to the sequence in which they were issued
by the processor."

So it provides no guarantee whatsoever wrt loads. However, reading
Documentation/memory-barriers.txt:

"Whilst they are technically interprocessor interaction considerations,
atomic operations are noted specially as some of them imply full memory
barriers and some don't, but they're very heavily relied on as a group
throughout the kernel." -> cmpxchg is part of them.

The same applies to the other atomic instructions we find in this list.
How is the correct ordering of loads wrt to cmxchg (and other atomic
ops) still ensured by this modification?

Thanks,

Mathieu

-- 
Mathieu Desnoyers
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68

Sparc64 support added to Userspace RCU

[Mathieu Desnoyers]

By the way, I just added sparc64 support to the Userspace RCU library,
available at: git://lttng.org/userspace-rcu.git

Interesting files concerning sparc64 are:

urcu/arch_sparc64.h
urcu/uatomic_arch_sparc64.h

Direct links:
http://www.lttng.org/cgi-bin/gitweb.cgi?p=userspace-rcu.git;a=blob;f=urcu/arch_sparc64.h;hb=HEAD
http://www.lttng.org/cgi-bin/gitweb.cgi?p=userspace-rcu.git;a=blob;f=urcu/uatomic_arch_sparc64.h;hb=HEAD

Please note that this code targets user-space, not kernel-space.

Feedback is welcome,

Thanks,

Mathieu

* Mathieu Desnoyers (mathieu.desnoyers@polymtl.ca) wrote:
> Hi David,
> 
> I took a look at the current sparc64 cmpxchg implementation in the Linux
> kernel and stumbled on this commit:
> 
> commit 293666b7a17cb7a389fc274980439212386a19c4
> Author: David S. Miller <davem@davemloft.net>
> Date:   Sat Nov 15 13:33:25 2008 -0800
> 
>     sparc64: Stop using memory barriers for atomics and locks.
>     
>     The kernel always executes in the TSO memory model now,
>     so none of this stuff is necessary any more.
>     
>     With helpful feedback from Nick Piggin.
>     
>     Signed-off-by: David S. Miller <davem@davemloft.net>
> 
> Reading p. 152 A.9 Compare and Swap, I see that the cas instruction does
> not imply a memory barrier.
> 
> Reading http://docs.sun.com/app/docs/doc/801-6678/6i11oelck?a=view
> 
> I see that:
> 
> "TSO guarantees that the store, FLUSH, and atomic load-store
> instructions of all processors appear to be executed by memory serially
> in a single order called the memory order. Furthermore, the sequence of
> store, FLUSH, and atomic load-store instructions in the memory order for
> a given processor is identical to the sequence in which they were issued
> by the processor."
> 
> So it provides no guarantee whatsoever wrt loads. However, reading
> Documentation/memory-barriers.txt:
> 
> "Whilst they are technically interprocessor interaction considerations,
> atomic operations are noted specially as some of them imply full memory
> barriers and some don't, but they're very heavily relied on as a group
> throughout the kernel." -> cmpxchg is part of them.
> 
> The same applies to the other atomic instructions we find in this list.
> How is the correct ordering of loads wrt to cmxchg (and other atomic
> ops) still ensured by this modification?
> 
> Thanks,
> 
> Mathieu
> 
> -- 
> Mathieu Desnoyers
> OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68

-- 
Mathieu Desnoyers
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68

Re: sparc64 cmpxchg is not a full memory barrier anymore ?

[David Miller]

From: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>
Date: Thu, 22 Oct 2009 14:32:42 -0400

> The same applies to the other atomic instructions we find in this list.
> How is the correct ordering of loads wrt to cmxchg (and other atomic
> ops) still ensured by this modification?

All actual sparc64 chips implement more strict ordering than
the V9 specification permits.  The memory barriers were just
nops and actually not doing anything more than the chip
already does for us.

Re: sparc64 cmpxchg is not a full memory barrier anymore ?

[Mathieu Desnoyers]

* David Miller (davem@davemloft.net) wrote:
> From: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>
> Date: Thu, 22 Oct 2009 14:32:42 -0400
> 
> > The same applies to the other atomic instructions we find in this list.
> > How is the correct ordering of loads wrt to cmxchg (and other atomic
> > ops) still ensured by this modification?
> 
> All actual sparc64 chips implement more strict ordering than
> the V9 specification permits.  The memory barriers were just
> nops and actually not doing anything more than the chip
> already does for us.

OK. Perhaps adding a comment to that effect near sparc mb()
implementation would be appropriate ?

Thanks,

Mathieu


-- 
Mathieu Desnoyers
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68

Re: sparc64 cmpxchg is not a full memory barrier anymore ?

[David Miller]

From: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>
Date: Fri, 23 Oct 2009 08:33:27 -0400

> OK. Perhaps adding a comment to that effect near sparc mb()
> implementation would be appropriate ?

Sure, I'll take care of that when I get back from Japan.

Re: Sparc64 support added to Userspace RCU

[David Miller]

From: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>
Date: Thu, 22 Oct 2009 15:57:53 -0400

> Feedback is welcome,

__sparc_v8__ does not mean 64-bit longs or pointers as implied here:

#ifndef __SIZEOF_LONG__
#if (defined(__sparc_v8__) || defined(__sparc_v9__))
#define __SIZEOF_LONG__ 8
#else
#define __SIZEOF_LONG__ 4
#endif
#endif

"v8" was the last 32-bit standard of the cpu architecture.

Otherwise looks fine.

Although this port won't be very useful until it can be built in
32-bit mode.  As most applications on sparc64 systems are still
32-bit.  Like on powerpc, people only rarely build things 64-bit on
sparc64.

You need to build with "-Wa,-Av9a" on the GCC command line for that
case so that the assembler allows the v9 instructions emitted by your
macros, even when building for 32-bit.

Hope this helps.

Re: Sparc64 support added to Userspace RCU

[Mathieu Desnoyers]

* David Miller (davem@davemloft.net) wrote:
> From: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>
> Date: Thu, 22 Oct 2009 15:57:53 -0400
> 
> > Feedback is welcome,
> 
> __sparc_v8__ does not mean 64-bit longs or pointers as implied here:
> 
> #ifndef __SIZEOF_LONG__
> #if (defined(__sparc_v8__) || defined(__sparc_v9__))
> #define __SIZEOF_LONG__ 8
> #else
> #define __SIZEOF_LONG__ 4
> #endif
> #endif
> 
> "v8" was the last 32-bit standard of the cpu architecture.
> 
> Otherwise looks fine.
> 
> Although this port won't be very useful until it can be built in
> 32-bit mode.  As most applications on sparc64 systems are still
> 32-bit.  Like on powerpc, people only rarely build things 64-bit on
> sparc64.
> 
> You need to build with "-Wa,-Av9a" on the GCC command line for that
> case so that the assembler allows the v9 instructions emitted by your
> macros, even when building for 32-bit.
> 
> Hope this helps.

Yes ! Thanks a lot for this information. I'll push the following fix in
my tree.

Mathieu

diff --git a/README b/README
index 883b808..ca23d43 100644
--- a/README
+++ b/README
@@ -18,11 +18,14 @@ BUILDING
 		Forcing a 32-bit build with 386 backward compatibility:
 		* CFLAGS="-m32 -g -O2" ./configure --host=i386-pc-linux-gnu
 
+		Forcing a 32-bit build for Sparcv9 (typical for Sparc v9)
+		* CFLAGS="-m32 -Wa,-Av9a -g -O2" ./configure
+
 ARCHITECTURES SUPPORTED
 -----------------------
 
 Currently, x86 (i386, i486, i586, i686), x86 64-bit, PowerPC 32/64, S390, S390x
-and Sparc64 are supported. Only tested on Linux so far, but should
+and Sparcv9 32/64 are supported. Only tested on Linux so far, but should
 theoretically work on other operating systems.
 
 QUICK START GUIDE
diff --git a/urcu/uatomic_arch_sparc64.h b/urcu/uatomic_arch_sparc64.h
index 35b9ac2..28829d2 100644
--- a/urcu/uatomic_arch_sparc64.h
+++ b/urcu/uatomic_arch_sparc64.h
@@ -24,7 +24,7 @@
 #include <urcu/system.h>
 
 #ifndef __SIZEOF_LONG__
-#if (defined(__sparc_v8__) || defined(__sparc_v9__))
+#ifdef __LP64__
 #define __SIZEOF_LONG__ 8
 #else
 #define __SIZEOF_LONG__ 4

-- 
Mathieu Desnoyers
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68