[patch 2/8] mutex subsystem, add asm-generic/mutex-[dec|xchg].h implementations

[patch 2/8] mutex subsystem, add asm-generic/mutex-[dec|xchg].h implementations

[Ingo Molnar]

add the two generic mutex fastpath implementations.

Signed-off-by: Arjan van de Ven <arjan@infradead.org>
Signed-off-by: Ingo Molnar <mingo@elte.hu>

----

 include/asm-generic/mutex-dec.h  |   66 ++++++++++++++++++++++++++++++++++++
 include/asm-generic/mutex-xchg.h |   71 +++++++++++++++++++++++++++++++++++++++
 2 files changed, 137 insertions(+)

Index: linux/include/asm-generic/mutex-dec.h
===================================================================
--- /dev/null
+++ linux/include/asm-generic/mutex-dec.h
@@ -0,0 +1,66 @@
+/*
+ * asm-generic/mutex-dec.h
+ *
+ * Generic wrappers for the mutex fastpath based on an xchg() implementation
+ *
+ */
+#ifndef _ASM_GENERIC_MUTEX_DEC_H
+#define _ASM_GENERIC_MUTEX_DEC_H
+
+/**
+ *  __mutex_fastpath_lock - try to take the lock by moving the count
+ *                          from 1 to a 0 value
+ *  @count: pointer of type atomic_t
+ *  @fn: function to call if the original value was not 1
+ *
+ * Change the count from 1 to a value lower than 1, and call <fn> if it
+ * wasn't 1 originally. This function MUST leave the value lower than 1
+ * even when the "1" assertion wasn't true.
+ */
+#define __mutex_fastpath_lock(count, fn_name)				\
+do {									\
+	if (unlikely(atomic_dec_return(count) < 0))			\
+		fn_name(count);						\
+} while (0)
+
+/**
+ *  __mutex_fastpath_lock_retval - try to take the lock by moving the count
+ *                                 from 1 to a 0 value
+ *  @count: pointer of type atomic_t
+ *  @fn: function to call if the original value was not 1
+ *
+ * Change the count from 1 to a value lower than 1, and call <fn> if it
+ * wasn't 1 originally. This function returns 0 if the fastpath succeeds,
+ * or anything the slow path function returns
+ */
+static inline int
+__mutex_fastpath_lock_retval(atomic_t *count,
+			     int (*fn_name)(atomic_t *))
+{
+	if (unlikely(atomic_dec_return(count) < 0))
+		return fn_name(count);
+	else
+		return 0;
+}
+
+/**
+ *  __mutex_fastpath_unlock - try to promote the mutex from 0 to 1
+ *  @count: pointer of type atomic_t
+ *  @fn: function to call if the original value was not 1
+ *
+ * try to promote the mutex from 0 to 1. if it wasn't 0, call <function>
+ * In the failure case, this function is allowed to either set the value to
+ * 1, or to set it to a value lower than one.
+ * If the implementation sets it to a value of lower than one, the
+ * __mutex_slowpath_needs_to_unlock() macro needs to return 1, it needs
+ * to return 0 otherwise.
+ */
+#define __mutex_fastpath_unlock(count, fn_name)				\
+do {									\
+	if (unlikely(atomic_inc_return(count) <= 0))			\
+		fn_name(count);						\
+} while (0)
+
+#define __mutex_slowpath_needs_to_unlock()		1
+
+#endif
Index: linux/include/asm-generic/mutex-xchg.h
===================================================================
--- /dev/null
+++ linux/include/asm-generic/mutex-xchg.h
@@ -0,0 +1,71 @@
+/*
+ * asm-generic/mutex-xchg.h
+ *
+ * Generic wrappers for the mutex fastpath based on an xchg() implementation
+ *
+ * NOTE: An xchg based implementation is less optimal than a
+ *       decrement/increment based implementation. If your architecture
+ *       has a reasonable atomic dec/inc then don't use this file, use
+ *       mutex-dec.h instead!
+ */
+#ifndef _ASM_GENERIC_MUTEX_XCHG_H
+#define _ASM_GENERIC_MUTEX_XCHG_H
+
+/**
+ *  __mutex_fastpath_lock - try to take the lock by moving the count
+ *                          from 1 to a 0 value
+ *  @count: pointer of type atomic_t
+ *  @fn: function to call if the original value was not 1
+ *
+ * Change the count from 1 to a value lower than 1, and call <fn> if it
+ * wasn't 1 originally. This function MUST leave the value lower than 1
+ * even when the "1" assertion wasn't true.
+ */
+#define __mutex_fastpath_lock(count, fn_name)				\
+do {									\
+	if (unlikely(atomic_xchg(count, 0) != 1))			\
+		fn_name(count);						\
+} while (0)
+
+
+/**
+ *  __mutex_fastpath_lock_retval - try to take the lock by moving the count
+ *                                 from 1 to a 0 value
+ *  @count: pointer of type atomic_t
+ *  @fn: function to call if the original value was not 1
+ *
+ * Change the count from 1 to a value lower than 1, and call <fn> if it
+ * wasn't 1 originally. This function returns 0 if the fastpath succeeds,
+ * or anything the slow path function returns
+ */
+static inline int
+__mutex_fastpath_lock_retval(atomic_t *count,
+			     int (*fn_name)(atomic_t *))
+{
+	if (unlikely(atomic_xchg(count, 0) != 1))
+		return fn_name(count);
+	else
+		return 0;
+}
+
+/**
+ *  __mutex_fastpath_unlock - try to promote the mutex from 0 to 1
+ *  @count: pointer of type atomic_t
+ *  @fn: function to call if the original value was not 1
+ *
+ * try to promote the mutex from 0 to 1. if it wasn't 0, call <function>
+ * In the failure case, this function is allowed to either set the value to
+ * 1, or to set it to a value lower than one.
+ * If the implementation sets it to a value of lower than one, the
+ * __mutex_slowpath_needs_to_unlock() macro needs to return 1, it needs
+ * to return 0 otherwise.
+ */
+#define __mutex_fastpath_unlock(count, fn_name)				\
+do {									\
+	if (unlikely(atomic_xchg(count, 1) != 0))			\
+		fn_name(count);						\
+} while (0)
+
+#define __mutex_slowpath_needs_to_unlock()		0
+
+#endif

Re: [patch 2/8] mutex subsystem, add asm-generic/mutex-[dec|xchg].h implementations

[Linus Torvalds]

On Fri, 23 Dec 2005, Ingo Molnar wrote:
>
> add the two generic mutex fastpath implementations.

Now this looks more like it. This is readable code without any #ifdef's in 
the middle.

Now the only #ifdef's seem to be for mutex debugging. Might it be 
worthwhile to have a generic debugging, that just uses spinlocks and just 
accept that it's going to be slow, but shared across absolutely all 
architectures?

Then you could have <linux/mutex.h> just doing a single

	#ifdef CONFIG_MUTEX_DEBUG
	# include <asm-generic/mutex-dbg.h>
	#else
	# include <asm/mutex.h>
	#endif

and have muted-dbg.h just contain prototypes (no point in inlining them, 
they're going to be big anyway) and then have a 

	obj$(CONFIG_MUTEX_DEBUG) += mutex-debug.c

in the kernel/ subdirectory? That way you could _really_ have a clean 
separation, with absolutely zero pollution of any architecture mess or 
debugging #ifdef's in any implementation code.

At that point I'd like to switch to mutexes just because the code is 
cleaner!

		Linus

Re: [patch 2/8] mutex subsystem, add asm-generic/mutex-[dec|xchg].h implementations

[Ingo Molnar]

* Linus Torvalds <torvalds@osdl.org> wrote:

> On Fri, 23 Dec 2005, Ingo Molnar wrote:
> >
> > add the two generic mutex fastpath implementations.
> 
> Now this looks more like it. This is readable code without any #ifdef's in 
> the middle.
> 
> Now the only #ifdef's seem to be for mutex debugging. Might it be 
> worthwhile to have a generic debugging, that just uses spinlocks and 
> just accept that it's going to be slow, but shared across absolutely 
> all architectures?

yeah, that's how it's working right now - the debugging variant still 
includes the arch header, but ignores it, and always does the slowpath.  
That's one of the advantages of the arch defining the fastpath, but not 
the API! Debugging needs to set up more state so it needs the spinlock 
all the time.

> 	obj$(CONFIG_MUTEX_DEBUG) += mutex-debug.c
> 
> in the kernel/ subdirectory? That way you could _really_ have a clean 
> separation, with absolutely zero pollution of any architecture mess or 
> debugging #ifdef's in any implementation code.

i think we are quite close to this already. We still want to share the 
slowpath because we want to debug it. I'll try to put the mutex-debug.c 
functions into a separate object file, but that introduces more 
interfaces between the two than i wanted to - even if debugging is slow, 
mutexes with full debugging are still faster in the VFS test than 
semaphores ;) I also wanted to introduce a lighter mode of debugging 
that could have the fastpath enabled, and which distributions could 
enable by default in their QA phase. (just like SPINLOCK_DEBUG)

	Ingo