Re: [PATCH v2 1/7] atomics: add acquire/release/relaxed variants of some atomic operations

Waiman Long
2015-07-17T00:07:32Z

On 07/16/2015 11:32 AM, Will Deacon wrote:
> Whilst porting the generic qrwlock code over to arm64, it became
> apparent that any portable locking code needs finer-grained control of
> the memory-ordering guarantees provided by our atomic routines.
>
> In particular: xchg, cmpxchg, {add,sub}_return are often used in
> situations where full barrier semantics (currently the only option
> available) are not required. For example, when a reader increments a
> reader count to obtain a lock, checking the old value to see if a writer
> was present, only acquire semantics are strictly needed.
>
> This patch introduces three new ordering semantics for these operations:
>
> - *_relaxed: No ordering guarantees. This is similar to what we have
> already for the non-return atomics (e.g. atomic_add).
>
> - *_acquire: ACQUIRE semantics, similar to smp_load_acquire.
>
> - *_release: RELEASE semantics, similar to smp_store_release.
>
> In memory-ordering speak, this means that the acquire/release semantics
> are RCpc as opposed to RCsc. Consequently a RELEASE followed by an
> ACQUIRE does not imply a full barrier, as already documented in
> memory-barriers.txt.
>
> Currently, all the new macros are conditionally mapped to the full-mb
> variants, however if the *_relaxed version is provided by the
> architecture, then the acquire/release variants are constructed by
> supplementing the relaxed routine with an explicit barrier.
>
> Cc: Peter Zijlstra<peterz@infradead.org>
> Signed-off-by: Will Deacon<will.deacon@arm.com>
> ---
> include/linux/atomic.h | 312 +++++++++++++++++++++++++++++++++++++++++++++++++
> 1 file changed, 312 insertions(+)
>
> diff --git a/include/linux/atomic.h b/include/linux/atomic.h
> index 5b08a8540ecf..08c2f6e56f76 100644
> --- a/include/linux/atomic.h
> +++ b/include/linux/atomic.h
> @@ -2,6 +2,318 @@
> #ifndef _LINUX_ATOMIC_H
> #define _LINUX_ATOMIC_H
> #include<asm/atomic.h>
> +#include<asm/barrier.h>
> +
> +/*
> + * Relaxed variants of xchg, cmpxchg and some atomic operations.
> + *
> + * We support four variants:
> + *
> + * - Fully ordered: The default implementation, no suffix required.
> + * - Acquire: Provides ACQUIRE semantics, _acquire suffix.
> + * - Release: Provides RELEASE semantics, _release suffix.
> + * - Relaxed: No ordering guarantees, _relaxed suffix.
> + *
> + * See Documentation/memory-barriers.txt for ACQUIRE/RELEASE definitions.
> + */
> +
> +#ifndef atomic_read_acquire
> +#define atomic_read_acquire(v) smp_load_acquire(&(v)->counter)
> +#endif
> +
> +#ifndef atomic_set_release
> +#define atomic_set_release(v, i) smp_store_release(&(v)->counter, (i))
> +#endif
> +
> +/*
> + * The idea here is to build acquire/release variants by adding explicit
> + * barriers on top of the relaxed variant. In the case where the relaxed
> + * variant is already fully ordered, no additional barriers are needed.
> + */
> +#define __atomic_op_acquire(ret_t, op, ...) \
> +({ \
> + ret_t __ret = op##_relaxed(__VA_ARGS__); \
> + smp_mb__after_atomic(); \
> + __ret; \
> +})
> +
> +#define __atomic_op_release(ret_t, op, ...) \
> +({ \
> + ret_t __ret; \
> + smp_mb__before_atomic(); \
> + __ret = op##_relaxed(__VA_ARGS__); \
> + __ret; \
> +})
> +
> +#define __atomic_op_fence(ret_t, op, ...) \
> +({ \
> + ret_t __ret; \
> + smp_mb__before_atomic(); \
> + __ret = op##_relaxed(__VA_ARGS__); \
> + smp_mb__after_atomic(); \
> + __ret; \
> +})
> +
> +#ifndef atomic_add_return_relaxed
> +#define atomic_add_return_relaxed atomic_add_return
> +#define atomic_add_return_acquire atomic_add_return
> +#define atomic_add_return_release atomic_add_return
> +
> +#else /* atomic_add_return_relaxed */
> +
> +#ifndef atomic_add_return_acquire
> +#define atomic_add_return_acquire(...) \
> + __atomic_op_acquire(int, atomic_add_return, __VA_ARGS__)
> +#endif
> +
> +#ifndef atomic_add_return_release
> +#define atomic_add_return_release(...) \
> + __atomic_op_release(int, atomic_add_return, __VA_ARGS__)
> +#endif
> +
> +#ifndef atomic_add_return
> +#define atomic_add_return(...) \
> + __atomic_op_fence(int, atomic_add_return, __VA_ARGS__)
> +#endif
> +#endif /* atomic_add_return_relaxed */
> +
> +#ifndef atomic_sub_return_relaxed
> +#define atomic_sub_return_relaxed atomic_sub_return
> +#define atomic_sub_return_acquire atomic_sub_return
> +#define atomic_sub_return_release atomic_sub_return
> +
> +#else /* atomic_sub_return_relaxed */
> +
> +#ifndef atomic_sub_return_acquire
> +#define atomic_sub_return_acquire(...) \
> + __atomic_op_acquire(int, atomic_sub_return, __VA_ARGS__)
> +#endif
> +
> +#ifndef atomic_sub_return_release
> +#define atomic_sub_return_release(...) \
> + __atomic_op_release(int, atomic_sub_return, __VA_ARGS__)
> +#endif
> +
> +#ifndef atomic_sub_return
> +#define atomic_sub_return(...) \
> + __atomic_op_fence(int, atomic_sub_return, __VA_ARGS__)
> +#endif
> +#endif /* atomic_sub_return_relaxed */
> +
> +#ifndef atomic_xchg_relaxed
> +#define atomic_xchg_relaxed atomic_xchg
> +#define atomic_xchg_acquire atomic_xchg
> +#define atomic_xchg_release atomic_xchg
> +
> +#else /* atomic_xchg_relaxed */
> +
> +#ifndef atomic_xchg_acquire
> +#define atomic_xchg_acquire(...) \
> + __atomic_op_acquire(int, atomic_xchg, __VA_ARGS__)
> +#endif
> +
> +#ifndef atomic_xchg_release
> +#define atomic_xchg_release(...) \
> + __atomic_op_release(int, atomic_xchg, __VA_ARGS__)
> +#endif
> +
> +#ifndef atomic_xchg
> +#define atomic_xchg(...) \
> + __atomic_op_fence(int, atomic_xchg, __VA_ARGS__)
> +#endif
> +#endif /* atomic_xchg_relaxed */
> +
> +#ifndef atomic_cmpxchg_relaxed
> +#define atomic_cmpxchg_relaxed atomic_cmpxchg
> +#define atomic_cmpxchg_acquire atomic_cmpxchg
> +#define atomic_cmpxchg_release atomic_cmpxchg
> +
> +#else /* atomic_cmpxchg_relaxed */
> +
> +#ifndef atomic_cmpxchg_acquire
> +#define atomic_cmpxchg_acquire(...) \
> + __atomic_op_acquire(int, atomic_cmpxchg, __VA_ARGS__)
> +#endif
> +
> +#ifndef atomic_cmpxchg_release
> +#define atomic_cmpxchg_release(...) \
> + __atomic_op_release(int, atomic_cmpxchg, __VA_ARGS__)
> +#endif
> +
> +#ifndef atomic_cmpxchg
> +#define atomic_cmpxchg(...) \
> + __atomic_op_fence(int, atomic_cmpxchg, __VA_ARGS__)
> +#endif
> +#endif /* atomic_cmpxchg_relaxed */
> +
> +#ifndef atomic64_read_acquire
> +#define atomic64_read_acquire(v) smp_load_acquire(&(v)->counter)
> +#endif
> +
> +#ifndef atomic64_set_release
> +#define atomic64_set_release(v, i) smp_store_release(&(v)->counter, (i))
> +#endif
> +
> +#ifndef atomic64_add_return_relaxed
> +#define atomic64_add_return_relaxed atomic64_add_return
> +#define atomic64_add_return_acquire atomic64_add_return
> +#define atomic64_add_return_release atomic64_add_return
> +
> +#else /* atomic64_add_return_relaxed */
> +
> +#ifndef atomic64_add_return_acquire
> +#define atomic64_add_return_acquire(...) \
> + __atomic_op_acquire(long long, atomic64_add_return, __VA_ARGS__)
> +#endif
> +
> +#ifndef atomic64_add_return_release
> +#define atomic64_add_return_release(...) \
> + __atomic_op_release(long long, atomic64_add_return, __VA_ARGS__)
> +#endif
> +
> +#ifndef atomic64_add_return
> +#define atomic64_add_return(...) \
> + __atomic_op_fence(long long, atomic64_add_return, __VA_ARGS__)
> +#endif
> +#endif /* atomic64_add_return_relaxed */
> +

I have a minor nit. The atomic_add_return block is repeated with
"s/atomic_add_return/.../". Perhaps some more comments to delineate the
blocks more visibly will make this patch easier to read.

Cheers,
Longman