* [PATCH v4 0/3] locking/rwsem: Rwsem rearchitecture part 0 [not found] <20190213220014.kxk4dGPy_-ynCjBbjBru6na4k4ZyVXKT0j8GelBVXk8@z> @ 2019-02-13 22:00 ` Waiman Long [not found] ` <20190213220015.1ysxH-7CNrdIFhN43HhpwNeF7RSBFoJQPWW5cLRyzjU@z> ` (3 more replies) 0 siblings, 4 replies; 15+ messages in thread From: Waiman Long @ 2019-02-13 22:00 UTC (permalink / raw) To: Peter Zijlstra, Ingo Molnar, Will Deacon, Thomas Gleixner Cc: linux-kernel, linux-alpha, linux-arm-kernel, linux-c6x-dev, uclinux-h8-devel, linux-hexagon, linux-ia64, linux-m68k, linux-mips, nios2-dev, openrisc, linux-parisc, linuxppc-dev, linux-riscv, linux-s390, linux-sh, sparclinux, linux-um, linux-xtensa, linux-arch, x86, Arnd Bergmann, Borislav Petkov, H. Peter Anvin, Davidlohr Bueso, Linus Torvalds, Andrew Morton, Tim Chen, Waiman Long v4: - Remove rwsem-spinlock.c and make all archs use rwsem-xadd.c. v3: - Optimize __down_read_trylock() for the uncontended case as suggested by Linus. v2: - Add patch 2 to optimize __down_read_trylock() as suggested by PeterZ. - Update performance test data in patch 1. The goal of this patchset is to remove the architecture specific files for rwsem-xadd to make it easer to add enhancements in the later rwsem patches. It also removes the legacy rwsem-spinlock.c file and make all the architectures use one single implementation of rwsem - rwsem-xadd.c. Waiman Long (3): locking/rwsem: Remove arch specific rwsem files locking/rwsem: Remove rwsem-spinlock.c & use rwsem-xadd.c for all archs locking/rwsem: Optimize down_read_trylock() MAINTAINERS | 1 - arch/alpha/Kconfig | 7 - arch/alpha/include/asm/rwsem.h | 211 ------------------------- arch/arc/Kconfig | 3 - arch/arm/Kconfig | 4 - arch/arm/include/asm/Kbuild | 1 - arch/arm64/Kconfig | 3 - arch/arm64/include/asm/Kbuild | 1 - arch/c6x/Kconfig | 3 - arch/csky/Kconfig | 3 - arch/h8300/Kconfig | 3 - arch/hexagon/Kconfig | 6 - arch/hexagon/include/asm/Kbuild | 1 - arch/ia64/Kconfig | 4 - arch/ia64/include/asm/rwsem.h | 172 -------------------- arch/m68k/Kconfig | 7 - arch/microblaze/Kconfig | 6 - arch/mips/Kconfig | 7 - arch/nds32/Kconfig | 3 - arch/nios2/Kconfig | 3 - arch/openrisc/Kconfig | 6 - arch/parisc/Kconfig | 6 - arch/powerpc/Kconfig | 7 - arch/powerpc/include/asm/Kbuild | 1 - arch/riscv/Kconfig | 3 - arch/s390/Kconfig | 6 - arch/s390/include/asm/Kbuild | 1 - arch/sh/Kconfig | 6 - arch/sh/include/asm/Kbuild | 1 - arch/sparc/Kconfig | 8 - arch/sparc/include/asm/Kbuild | 1 - arch/unicore32/Kconfig | 6 - arch/x86/Kconfig | 3 - arch/x86/include/asm/rwsem.h | 237 ---------------------------- arch/x86/lib/Makefile | 1 - arch/x86/lib/rwsem.S | 156 ------------------ arch/x86/um/Kconfig | 6 - arch/x86/um/Makefile | 1 - arch/xtensa/Kconfig | 3 - arch/xtensa/include/asm/Kbuild | 1 - include/asm-generic/rwsem.h | 140 ----------------- include/linux/rwsem-spinlock.h | 47 ------ include/linux/rwsem.h | 9 +- kernel/Kconfig.locks | 2 +- kernel/locking/Makefile | 4 +- kernel/locking/percpu-rwsem.c | 2 + kernel/locking/rwsem-spinlock.c | 339 ---------------------------------------- kernel/locking/rwsem.h | 130 +++++++++++++++ 48 files changed, 135 insertions(+), 1447 deletions(-) delete mode 100644 arch/alpha/include/asm/rwsem.h delete mode 100644 arch/ia64/include/asm/rwsem.h delete mode 100644 arch/x86/include/asm/rwsem.h delete mode 100644 arch/x86/lib/rwsem.S delete mode 100644 include/asm-generic/rwsem.h delete mode 100644 include/linux/rwsem-spinlock.h delete mode 100644 kernel/locking/rwsem-spinlock.c -- 1.8.3.1 ^ permalink raw reply [flat|nested] 15+ messages in thread
[parent not found: <20190213220015.1ysxH-7CNrdIFhN43HhpwNeF7RSBFoJQPWW5cLRyzjU@z>]
* [PATCH v4 1/3] locking/rwsem: Remove arch specific rwsem files [not found] ` <20190213220015.1ysxH-7CNrdIFhN43HhpwNeF7RSBFoJQPWW5cLRyzjU@z> @ 2019-02-13 22:00 ` Waiman Long 2019-02-20 14:47 ` Will Deacon 0 siblings, 1 reply; 15+ messages in thread From: Waiman Long @ 2019-02-13 22:00 UTC (permalink / raw) To: Peter Zijlstra, Ingo Molnar, Will Deacon, Thomas Gleixner Cc: linux-kernel, linux-alpha, linux-arm-kernel, linux-c6x-dev, uclinux-h8-devel, linux-hexagon, linux-ia64, linux-m68k, linux-mips, nios2-dev, openrisc, linux-parisc, linuxppc-dev, linux-riscv, linux-s390, linux-sh, sparclinux, linux-um, linux-xtensa, linux-arch, x86, Arnd Bergmann, Borislav Petkov, H. Peter Anvin, Davidlohr Bueso, Linus Torvalds, Andrew Morton, Tim Chen, Waiman Long As the generic rwsem-xadd code is using the appropriate acquire and release versions of the atomic operations, the arch specific rwsem.h files will not be that much faster than the generic code as long as the atomic functions are properly implemented. So we can remove those arch specific rwsem.h and stop building asm/rwsem.h to reduce maintenance effort. Currently, only x86, alpha and ia64 have implemented architecture specific fast paths. I don't have access to alpha and ia64 systems for testing, but they are legacy systems that are not likely to be updated to the latest kernel anyway. By using a rwsem microbenchmark, the total locking rates on a 4-socket 56-core 112-thread x86-64 system before and after the patch were as follows (mixed means equal # of read and write locks): Before Patch After Patch # of Threads wlock rlock mixed wlock rlock mixed ------------ ----- ----- ----- ----- ----- ----- 1 29,201 30,143 29,458 28,615 30,172 29,201 2 6,807 13,299 1,171 7,725 15,025 1,804 4 6,504 12,755 1,520 7,127 14,286 1,345 8 6,762 13,412 764 6,826 13,652 726 16 6,693 15,408 662 6,599 15,938 626 32 6,145 15,286 496 5,549 15,487 511 64 5,812 15,495 60 5,858 15,572 60 There were some run-to-run variations for the multi-thread tests. For x86-64, using the generic C code fast path seems to be a little bit faster than the assembly version with low lock contention. Looking at the assembly version of the fast paths, there are assembly to/from C code wrappers that save and restore all the callee-clobbered registers (7 registers on x86-64). The assembly generated from the generic C code doesn't need to do that. That may explain the slight performance gain here. The generic asm rwsem.h can also be merged into kernel/locking/rwsem.h with no code change as no other code other than those under kernel/locking needs to access the internal rwsem macros and functions. Signed-off-by: Waiman Long <longman@redhat.com> --- MAINTAINERS | 1 - arch/alpha/include/asm/rwsem.h | 211 ----------------------------------- arch/arm/include/asm/Kbuild | 1 - arch/arm64/include/asm/Kbuild | 1 - arch/hexagon/include/asm/Kbuild | 1 - arch/ia64/include/asm/rwsem.h | 172 ----------------------------- arch/powerpc/include/asm/Kbuild | 1 - arch/s390/include/asm/Kbuild | 1 - arch/sh/include/asm/Kbuild | 1 - arch/sparc/include/asm/Kbuild | 1 - arch/x86/include/asm/rwsem.h | 237 ---------------------------------------- arch/x86/lib/Makefile | 1 - arch/x86/lib/rwsem.S | 156 -------------------------- arch/x86/um/Makefile | 1 - arch/xtensa/include/asm/Kbuild | 1 - include/asm-generic/rwsem.h | 140 ------------------------ include/linux/rwsem.h | 4 +- kernel/locking/percpu-rwsem.c | 2 + kernel/locking/rwsem.h | 130 ++++++++++++++++++++++ 19 files changed, 133 insertions(+), 930 deletions(-) delete mode 100644 arch/alpha/include/asm/rwsem.h delete mode 100644 arch/ia64/include/asm/rwsem.h delete mode 100644 arch/x86/include/asm/rwsem.h delete mode 100644 arch/x86/lib/rwsem.S delete mode 100644 include/asm-generic/rwsem.h diff --git a/MAINTAINERS b/MAINTAINERS index 9919840..053f536 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -8926,7 +8926,6 @@ F: arch/*/include/asm/spinlock*.h F: include/linux/rwlock*.h F: include/linux/mutex*.h F: include/linux/rwsem*.h -F: arch/*/include/asm/rwsem.h F: include/linux/seqlock.h F: lib/locking*.[ch] F: kernel/locking/ diff --git a/arch/alpha/include/asm/rwsem.h b/arch/alpha/include/asm/rwsem.h deleted file mode 100644 index cf8fc8f9..0000000 --- a/arch/alpha/include/asm/rwsem.h +++ /dev/null @@ -1,211 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0 */ -#ifndef _ALPHA_RWSEM_H -#define _ALPHA_RWSEM_H - -/* - * Written by Ivan Kokshaysky <ink@jurassic.park.msu.ru>, 2001. - * Based on asm-alpha/semaphore.h and asm-i386/rwsem.h - */ - -#ifndef _LINUX_RWSEM_H -#error "please don't include asm/rwsem.h directly, use linux/rwsem.h instead" -#endif - -#ifdef __KERNEL__ - -#include <linux/compiler.h> - -#define RWSEM_UNLOCKED_VALUE 0x0000000000000000L -#define RWSEM_ACTIVE_BIAS 0x0000000000000001L -#define RWSEM_ACTIVE_MASK 0x00000000ffffffffL -#define RWSEM_WAITING_BIAS (-0x0000000100000000L) -#define RWSEM_ACTIVE_READ_BIAS RWSEM_ACTIVE_BIAS -#define RWSEM_ACTIVE_WRITE_BIAS (RWSEM_WAITING_BIAS + RWSEM_ACTIVE_BIAS) - -static inline int ___down_read(struct rw_semaphore *sem) -{ - long oldcount; -#ifndef CONFIG_SMP - oldcount = sem->count.counter; - sem->count.counter += RWSEM_ACTIVE_READ_BIAS; -#else - long temp; - __asm__ __volatile__( - "1: ldq_l %0,%1\n" - " addq %0,%3,%2\n" - " stq_c %2,%1\n" - " beq %2,2f\n" - " mb\n" - ".subsection 2\n" - "2: br 1b\n" - ".previous" - :"=&r" (oldcount), "=m" (sem->count), "=&r" (temp) - :"Ir" (RWSEM_ACTIVE_READ_BIAS), "m" (sem->count) : "memory"); -#endif - return (oldcount < 0); -} - -static inline void __down_read(struct rw_semaphore *sem) -{ - if (unlikely(___down_read(sem))) - rwsem_down_read_failed(sem); -} - -static inline int __down_read_killable(struct rw_semaphore *sem) -{ - if (unlikely(___down_read(sem))) - if (IS_ERR(rwsem_down_read_failed_killable(sem))) - return -EINTR; - - return 0; -} - -/* - * trylock for reading -- returns 1 if successful, 0 if contention - */ -static inline int __down_read_trylock(struct rw_semaphore *sem) -{ - long old, new, res; - - res = atomic_long_read(&sem->count); - do { - new = res + RWSEM_ACTIVE_READ_BIAS; - if (new <= 0) - break; - old = res; - res = atomic_long_cmpxchg(&sem->count, old, new); - } while (res != old); - return res >= 0 ? 1 : 0; -} - -static inline long ___down_write(struct rw_semaphore *sem) -{ - long oldcount; -#ifndef CONFIG_SMP - oldcount = sem->count.counter; - sem->count.counter += RWSEM_ACTIVE_WRITE_BIAS; -#else - long temp; - __asm__ __volatile__( - "1: ldq_l %0,%1\n" - " addq %0,%3,%2\n" - " stq_c %2,%1\n" - " beq %2,2f\n" - " mb\n" - ".subsection 2\n" - "2: br 1b\n" - ".previous" - :"=&r" (oldcount), "=m" (sem->count), "=&r" (temp) - :"Ir" (RWSEM_ACTIVE_WRITE_BIAS), "m" (sem->count) : "memory"); -#endif - return oldcount; -} - -static inline void __down_write(struct rw_semaphore *sem) -{ - if (unlikely(___down_write(sem))) - rwsem_down_write_failed(sem); -} - -static inline int __down_write_killable(struct rw_semaphore *sem) -{ - if (unlikely(___down_write(sem))) { - if (IS_ERR(rwsem_down_write_failed_killable(sem))) - return -EINTR; - } - - return 0; -} - -/* - * trylock for writing -- returns 1 if successful, 0 if contention - */ -static inline int __down_write_trylock(struct rw_semaphore *sem) -{ - long ret = atomic_long_cmpxchg(&sem->count, RWSEM_UNLOCKED_VALUE, - RWSEM_ACTIVE_WRITE_BIAS); - if (ret == RWSEM_UNLOCKED_VALUE) - return 1; - return 0; -} - -static inline void __up_read(struct rw_semaphore *sem) -{ - long oldcount; -#ifndef CONFIG_SMP - oldcount = sem->count.counter; - sem->count.counter -= RWSEM_ACTIVE_READ_BIAS; -#else - long temp; - __asm__ __volatile__( - " mb\n" - "1: ldq_l %0,%1\n" - " subq %0,%3,%2\n" - " stq_c %2,%1\n" - " beq %2,2f\n" - ".subsection 2\n" - "2: br 1b\n" - ".previous" - :"=&r" (oldcount), "=m" (sem->count), "=&r" (temp) - :"Ir" (RWSEM_ACTIVE_READ_BIAS), "m" (sem->count) : "memory"); -#endif - if (unlikely(oldcount < 0)) - if ((int)oldcount - RWSEM_ACTIVE_READ_BIAS == 0) - rwsem_wake(sem); -} - -static inline void __up_write(struct rw_semaphore *sem) -{ - long count; -#ifndef CONFIG_SMP - sem->count.counter -= RWSEM_ACTIVE_WRITE_BIAS; - count = sem->count.counter; -#else - long temp; - __asm__ __volatile__( - " mb\n" - "1: ldq_l %0,%1\n" - " subq %0,%3,%2\n" - " stq_c %2,%1\n" - " beq %2,2f\n" - " subq %0,%3,%0\n" - ".subsection 2\n" - "2: br 1b\n" - ".previous" - :"=&r" (count), "=m" (sem->count), "=&r" (temp) - :"Ir" (RWSEM_ACTIVE_WRITE_BIAS), "m" (sem->count) : "memory"); -#endif - if (unlikely(count)) - if ((int)count == 0) - rwsem_wake(sem); -} - -/* - * downgrade write lock to read lock - */ -static inline void __downgrade_write(struct rw_semaphore *sem) -{ - long oldcount; -#ifndef CONFIG_SMP - oldcount = sem->count.counter; - sem->count.counter -= RWSEM_WAITING_BIAS; -#else - long temp; - __asm__ __volatile__( - "1: ldq_l %0,%1\n" - " addq %0,%3,%2\n" - " stq_c %2,%1\n" - " beq %2,2f\n" - " mb\n" - ".subsection 2\n" - "2: br 1b\n" - ".previous" - :"=&r" (oldcount), "=m" (sem->count), "=&r" (temp) - :"Ir" (-RWSEM_WAITING_BIAS), "m" (sem->count) : "memory"); -#endif - if (unlikely(oldcount < 0)) - rwsem_downgrade_wake(sem); -} - -#endif /* __KERNEL__ */ -#endif /* _ALPHA_RWSEM_H */ diff --git a/arch/arm/include/asm/Kbuild b/arch/arm/include/asm/Kbuild index 1d66db9..989e1a7 100644 --- a/arch/arm/include/asm/Kbuild +++ b/arch/arm/include/asm/Kbuild @@ -12,7 +12,6 @@ generic-y += mm-arch-hooks.h generic-y += msi.h generic-y += parport.h generic-y += preempt.h -generic-y += rwsem.h generic-y += seccomp.h generic-y += segment.h generic-y += serial.h diff --git a/arch/arm64/include/asm/Kbuild b/arch/arm64/include/asm/Kbuild index 1e17ea5..60a933b 100644 --- a/arch/arm64/include/asm/Kbuild +++ b/arch/arm64/include/asm/Kbuild @@ -16,7 +16,6 @@ generic-y += mm-arch-hooks.h generic-y += msi.h generic-y += qrwlock.h generic-y += qspinlock.h -generic-y += rwsem.h generic-y += segment.h generic-y += serial.h generic-y += set_memory.h diff --git a/arch/hexagon/include/asm/Kbuild b/arch/hexagon/include/asm/Kbuild index b25fd42..0a12feb 100644 --- a/arch/hexagon/include/asm/Kbuild +++ b/arch/hexagon/include/asm/Kbuild @@ -26,7 +26,6 @@ generic-y += mm-arch-hooks.h generic-y += pci.h generic-y += percpu.h generic-y += preempt.h -generic-y += rwsem.h generic-y += sections.h generic-y += segment.h generic-y += serial.h diff --git a/arch/ia64/include/asm/rwsem.h b/arch/ia64/include/asm/rwsem.h deleted file mode 100644 index 9179106..0000000 --- a/arch/ia64/include/asm/rwsem.h +++ /dev/null @@ -1,172 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0 */ -/* - * R/W semaphores for ia64 - * - * Copyright (C) 2003 Ken Chen <kenneth.w.chen@intel.com> - * Copyright (C) 2003 Asit Mallick <asit.k.mallick@intel.com> - * Copyright (C) 2005 Christoph Lameter <cl@linux.com> - * - * Based on asm-i386/rwsem.h and other architecture implementation. - * - * The MSW of the count is the negated number of active writers and - * waiting lockers, and the LSW is the total number of active locks. - * - * The lock count is initialized to 0 (no active and no waiting lockers). - * - * When a writer subtracts WRITE_BIAS, it'll get 0xffffffff00000001 for - * the case of an uncontended lock. Readers increment by 1 and see a positive - * value when uncontended, negative if there are writers (and maybe) readers - * waiting (in which case it goes to sleep). - */ - -#ifndef _ASM_IA64_RWSEM_H -#define _ASM_IA64_RWSEM_H - -#ifndef _LINUX_RWSEM_H -#error "Please don't include <asm/rwsem.h> directly, use <linux/rwsem.h> instead." -#endif - -#include <asm/intrinsics.h> - -#define RWSEM_UNLOCKED_VALUE __IA64_UL_CONST(0x0000000000000000) -#define RWSEM_ACTIVE_BIAS (1L) -#define RWSEM_ACTIVE_MASK (0xffffffffL) -#define RWSEM_WAITING_BIAS (-0x100000000L) -#define RWSEM_ACTIVE_READ_BIAS RWSEM_ACTIVE_BIAS -#define RWSEM_ACTIVE_WRITE_BIAS (RWSEM_WAITING_BIAS + RWSEM_ACTIVE_BIAS) - -/* - * lock for reading - */ -static inline int -___down_read (struct rw_semaphore *sem) -{ - long result = ia64_fetchadd8_acq((unsigned long *)&sem->count.counter, 1); - - return (result < 0); -} - -static inline void -__down_read (struct rw_semaphore *sem) -{ - if (___down_read(sem)) - rwsem_down_read_failed(sem); -} - -static inline int -__down_read_killable (struct rw_semaphore *sem) -{ - if (___down_read(sem)) - if (IS_ERR(rwsem_down_read_failed_killable(sem))) - return -EINTR; - - return 0; -} - -/* - * lock for writing - */ -static inline long -___down_write (struct rw_semaphore *sem) -{ - long old, new; - - do { - old = atomic_long_read(&sem->count); - new = old + RWSEM_ACTIVE_WRITE_BIAS; - } while (atomic_long_cmpxchg_acquire(&sem->count, old, new) != old); - - return old; -} - -static inline void -__down_write (struct rw_semaphore *sem) -{ - if (___down_write(sem)) - rwsem_down_write_failed(sem); -} - -static inline int -__down_write_killable (struct rw_semaphore *sem) -{ - if (___down_write(sem)) { - if (IS_ERR(rwsem_down_write_failed_killable(sem))) - return -EINTR; - } - - return 0; -} - -/* - * unlock after reading - */ -static inline void -__up_read (struct rw_semaphore *sem) -{ - long result = ia64_fetchadd8_rel((unsigned long *)&sem->count.counter, -1); - - if (result < 0 && (--result & RWSEM_ACTIVE_MASK) == 0) - rwsem_wake(sem); -} - -/* - * unlock after writing - */ -static inline void -__up_write (struct rw_semaphore *sem) -{ - long old, new; - - do { - old = atomic_long_read(&sem->count); - new = old - RWSEM_ACTIVE_WRITE_BIAS; - } while (atomic_long_cmpxchg_release(&sem->count, old, new) != old); - - if (new < 0 && (new & RWSEM_ACTIVE_MASK) == 0) - rwsem_wake(sem); -} - -/* - * trylock for reading -- returns 1 if successful, 0 if contention - */ -static inline int -__down_read_trylock (struct rw_semaphore *sem) -{ - long tmp; - while ((tmp = atomic_long_read(&sem->count)) >= 0) { - if (tmp == atomic_long_cmpxchg_acquire(&sem->count, tmp, tmp+1)) { - return 1; - } - } - return 0; -} - -/* - * trylock for writing -- returns 1 if successful, 0 if contention - */ -static inline int -__down_write_trylock (struct rw_semaphore *sem) -{ - long tmp = atomic_long_cmpxchg_acquire(&sem->count, - RWSEM_UNLOCKED_VALUE, RWSEM_ACTIVE_WRITE_BIAS); - return tmp == RWSEM_UNLOCKED_VALUE; -} - -/* - * downgrade write lock to read lock - */ -static inline void -__downgrade_write (struct rw_semaphore *sem) -{ - long old, new; - - do { - old = atomic_long_read(&sem->count); - new = old - RWSEM_WAITING_BIAS; - } while (atomic_long_cmpxchg_release(&sem->count, old, new) != old); - - if (old < 0) - rwsem_downgrade_wake(sem); -} - -#endif /* _ASM_IA64_RWSEM_H */ diff --git a/arch/powerpc/include/asm/Kbuild b/arch/powerpc/include/asm/Kbuild index 77ff7fb..75bd77a 100644 --- a/arch/powerpc/include/asm/Kbuild +++ b/arch/powerpc/include/asm/Kbuild @@ -9,6 +9,5 @@ generic-y += irq_work.h generic-y += local64.h generic-y += mcs_spinlock.h generic-y += preempt.h -generic-y += rwsem.h generic-y += vtime.h generic-y += msi.h diff --git a/arch/s390/include/asm/Kbuild b/arch/s390/include/asm/Kbuild index e323977..4b77b42 100644 --- a/arch/s390/include/asm/Kbuild +++ b/arch/s390/include/asm/Kbuild @@ -21,7 +21,6 @@ generic-y += local64.h generic-y += mcs_spinlock.h generic-y += mm-arch-hooks.h generic-y += preempt.h -generic-y += rwsem.h generic-y += trace_clock.h generic-y += unaligned.h generic-y += word-at-a-time.h diff --git a/arch/sh/include/asm/Kbuild b/arch/sh/include/asm/Kbuild index a6ef3fe..b89fce4 100644 --- a/arch/sh/include/asm/Kbuild +++ b/arch/sh/include/asm/Kbuild @@ -16,7 +16,6 @@ generic-y += mm-arch-hooks.h generic-y += parport.h generic-y += percpu.h generic-y += preempt.h -generic-y += rwsem.h generic-y += serial.h generic-y += sizes.h generic-y += trace_clock.h diff --git a/arch/sparc/include/asm/Kbuild b/arch/sparc/include/asm/Kbuild index b82f64e..e843fc0 100644 --- a/arch/sparc/include/asm/Kbuild +++ b/arch/sparc/include/asm/Kbuild @@ -17,7 +17,6 @@ generic-y += mm-arch-hooks.h generic-y += module.h generic-y += msi.h generic-y += preempt.h -generic-y += rwsem.h generic-y += serial.h generic-y += trace_clock.h generic-y += word-at-a-time.h diff --git a/arch/x86/include/asm/rwsem.h b/arch/x86/include/asm/rwsem.h deleted file mode 100644 index 4c25cf6..0000000 --- a/arch/x86/include/asm/rwsem.h +++ /dev/null @@ -1,237 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0 */ -/* rwsem.h: R/W semaphores implemented using XADD/CMPXCHG for i486+ - * - * Written by David Howells (dhowells@redhat.com). - * - * Derived from asm-x86/semaphore.h - * - * - * The MSW of the count is the negated number of active writers and waiting - * lockers, and the LSW is the total number of active locks - * - * The lock count is initialized to 0 (no active and no waiting lockers). - * - * When a writer subtracts WRITE_BIAS, it'll get 0xffff0001 for the case of an - * uncontended lock. This can be determined because XADD returns the old value. - * Readers increment by 1 and see a positive value when uncontended, negative - * if there are writers (and maybe) readers waiting (in which case it goes to - * sleep). - * - * The value of WAITING_BIAS supports up to 32766 waiting processes. This can - * be extended to 65534 by manually checking the whole MSW rather than relying - * on the S flag. - * - * The value of ACTIVE_BIAS supports up to 65535 active processes. - * - * This should be totally fair - if anything is waiting, a process that wants a - * lock will go to the back of the queue. When the currently active lock is - * released, if there's a writer at the front of the queue, then that and only - * that will be woken up; if there's a bunch of consecutive readers at the - * front, then they'll all be woken up, but no other readers will be. - */ - -#ifndef _ASM_X86_RWSEM_H -#define _ASM_X86_RWSEM_H - -#ifndef _LINUX_RWSEM_H -#error "please don't include asm/rwsem.h directly, use linux/rwsem.h instead" -#endif - -#ifdef __KERNEL__ -#include <asm/asm.h> - -/* - * The bias values and the counter type limits the number of - * potential readers/writers to 32767 for 32 bits and 2147483647 - * for 64 bits. - */ - -#ifdef CONFIG_X86_64 -# define RWSEM_ACTIVE_MASK 0xffffffffL -#else -# define RWSEM_ACTIVE_MASK 0x0000ffffL -#endif - -#define RWSEM_UNLOCKED_VALUE 0x00000000L -#define RWSEM_ACTIVE_BIAS 0x00000001L -#define RWSEM_WAITING_BIAS (-RWSEM_ACTIVE_MASK-1) -#define RWSEM_ACTIVE_READ_BIAS RWSEM_ACTIVE_BIAS -#define RWSEM_ACTIVE_WRITE_BIAS (RWSEM_WAITING_BIAS + RWSEM_ACTIVE_BIAS) - -/* - * lock for reading - */ -#define ____down_read(sem, slow_path) \ -({ \ - struct rw_semaphore* ret; \ - asm volatile("# beginning down_read\n\t" \ - LOCK_PREFIX _ASM_INC "(%[sem])\n\t" \ - /* adds 0x00000001 */ \ - " jns 1f\n" \ - " call " slow_path "\n" \ - "1:\n\t" \ - "# ending down_read\n\t" \ - : "+m" (sem->count), "=a" (ret), \ - ASM_CALL_CONSTRAINT \ - : [sem] "a" (sem) \ - : "memory", "cc"); \ - ret; \ -}) - -static inline void __down_read(struct rw_semaphore *sem) -{ - ____down_read(sem, "call_rwsem_down_read_failed"); -} - -static inline int __down_read_killable(struct rw_semaphore *sem) -{ - if (IS_ERR(____down_read(sem, "call_rwsem_down_read_failed_killable"))) - return -EINTR; - return 0; -} - -/* - * trylock for reading -- returns 1 if successful, 0 if contention - */ -static inline bool __down_read_trylock(struct rw_semaphore *sem) -{ - long result, tmp; - asm volatile("# beginning __down_read_trylock\n\t" - " mov %[count],%[result]\n\t" - "1:\n\t" - " mov %[result],%[tmp]\n\t" - " add %[inc],%[tmp]\n\t" - " jle 2f\n\t" - LOCK_PREFIX " cmpxchg %[tmp],%[count]\n\t" - " jnz 1b\n\t" - "2:\n\t" - "# ending __down_read_trylock\n\t" - : [count] "+m" (sem->count), [result] "=&a" (result), - [tmp] "=&r" (tmp) - : [inc] "i" (RWSEM_ACTIVE_READ_BIAS) - : "memory", "cc"); - return result >= 0; -} - -/* - * lock for writing - */ -#define ____down_write(sem, slow_path) \ -({ \ - long tmp; \ - struct rw_semaphore* ret; \ - \ - asm volatile("# beginning down_write\n\t" \ - LOCK_PREFIX " xadd %[tmp],(%[sem])\n\t" \ - /* adds 0xffff0001, returns the old value */ \ - " test " __ASM_SEL(%w1,%k1) "," __ASM_SEL(%w1,%k1) "\n\t" \ - /* was the active mask 0 before? */\ - " jz 1f\n" \ - " call " slow_path "\n" \ - "1:\n" \ - "# ending down_write" \ - : "+m" (sem->count), [tmp] "=d" (tmp), \ - "=a" (ret), ASM_CALL_CONSTRAINT \ - : [sem] "a" (sem), "[tmp]" (RWSEM_ACTIVE_WRITE_BIAS) \ - : "memory", "cc"); \ - ret; \ -}) - -static inline void __down_write(struct rw_semaphore *sem) -{ - ____down_write(sem, "call_rwsem_down_write_failed"); -} - -static inline int __down_write_killable(struct rw_semaphore *sem) -{ - if (IS_ERR(____down_write(sem, "call_rwsem_down_write_failed_killable"))) - return -EINTR; - - return 0; -} - -/* - * trylock for writing -- returns 1 if successful, 0 if contention - */ -static inline bool __down_write_trylock(struct rw_semaphore *sem) -{ - bool result; - long tmp0, tmp1; - asm volatile("# beginning __down_write_trylock\n\t" - " mov %[count],%[tmp0]\n\t" - "1:\n\t" - " test " __ASM_SEL(%w1,%k1) "," __ASM_SEL(%w1,%k1) "\n\t" - /* was the active mask 0 before? */ - " jnz 2f\n\t" - " mov %[tmp0],%[tmp1]\n\t" - " add %[inc],%[tmp1]\n\t" - LOCK_PREFIX " cmpxchg %[tmp1],%[count]\n\t" - " jnz 1b\n\t" - "2:\n\t" - CC_SET(e) - "# ending __down_write_trylock\n\t" - : [count] "+m" (sem->count), [tmp0] "=&a" (tmp0), - [tmp1] "=&r" (tmp1), CC_OUT(e) (result) - : [inc] "er" (RWSEM_ACTIVE_WRITE_BIAS) - : "memory"); - return result; -} - -/* - * unlock after reading - */ -static inline void __up_read(struct rw_semaphore *sem) -{ - long tmp; - asm volatile("# beginning __up_read\n\t" - LOCK_PREFIX " xadd %[tmp],(%[sem])\n\t" - /* subtracts 1, returns the old value */ - " jns 1f\n\t" - " call call_rwsem_wake\n" /* expects old value in %edx */ - "1:\n" - "# ending __up_read\n" - : "+m" (sem->count), [tmp] "=d" (tmp) - : [sem] "a" (sem), "[tmp]" (-RWSEM_ACTIVE_READ_BIAS) - : "memory", "cc"); -} - -/* - * unlock after writing - */ -static inline void __up_write(struct rw_semaphore *sem) -{ - long tmp; - asm volatile("# beginning __up_write\n\t" - LOCK_PREFIX " xadd %[tmp],(%[sem])\n\t" - /* subtracts 0xffff0001, returns the old value */ - " jns 1f\n\t" - " call call_rwsem_wake\n" /* expects old value in %edx */ - "1:\n\t" - "# ending __up_write\n" - : "+m" (sem->count), [tmp] "=d" (tmp) - : [sem] "a" (sem), "[tmp]" (-RWSEM_ACTIVE_WRITE_BIAS) - : "memory", "cc"); -} - -/* - * downgrade write lock to read lock - */ -static inline void __downgrade_write(struct rw_semaphore *sem) -{ - asm volatile("# beginning __downgrade_write\n\t" - LOCK_PREFIX _ASM_ADD "%[inc],(%[sem])\n\t" - /* - * transitions 0xZZZZ0001 -> 0xYYYY0001 (i386) - * 0xZZZZZZZZ00000001 -> 0xYYYYYYYY00000001 (x86_64) - */ - " jns 1f\n\t" - " call call_rwsem_downgrade_wake\n" - "1:\n\t" - "# ending __downgrade_write\n" - : "+m" (sem->count) - : [sem] "a" (sem), [inc] "er" (-RWSEM_WAITING_BIAS) - : "memory", "cc"); -} - -#endif /* __KERNEL__ */ -#endif /* _ASM_X86_RWSEM_H */ diff --git a/arch/x86/lib/Makefile b/arch/x86/lib/Makefile index 140e618..9866520 100644 --- a/arch/x86/lib/Makefile +++ b/arch/x86/lib/Makefile @@ -23,7 +23,6 @@ obj-$(CONFIG_SMP) += msr-smp.o cache-smp.o lib-y := delay.o misc.o cmdline.o cpu.o lib-y += usercopy_$(BITS).o usercopy.o getuser.o putuser.o lib-y += memcpy_$(BITS).o -lib-$(CONFIG_RWSEM_XCHGADD_ALGORITHM) += rwsem.o lib-$(CONFIG_INSTRUCTION_DECODER) += insn.o inat.o insn-eval.o lib-$(CONFIG_RANDOMIZE_BASE) += kaslr.o lib-$(CONFIG_FUNCTION_ERROR_INJECTION) += error-inject.o diff --git a/arch/x86/lib/rwsem.S b/arch/x86/lib/rwsem.S deleted file mode 100644 index dc2ab6e..0000000 --- a/arch/x86/lib/rwsem.S +++ /dev/null @@ -1,156 +0,0 @@ -/* - * x86 semaphore implementation. - * - * (C) Copyright 1999 Linus Torvalds - * - * Portions Copyright 1999 Red Hat, Inc. - * - * This program is free software; you can redistribute it and/or - * modify it under the terms of the GNU General Public License - * as published by the Free Software Foundation; either version - * 2 of the License, or (at your option) any later version. - * - * rw semaphores implemented November 1999 by Benjamin LaHaise <bcrl@kvack.org> - */ - -#include <linux/linkage.h> -#include <asm/alternative-asm.h> -#include <asm/frame.h> - -#define __ASM_HALF_REG(reg) __ASM_SEL(reg, e##reg) -#define __ASM_HALF_SIZE(inst) __ASM_SEL(inst##w, inst##l) - -#ifdef CONFIG_X86_32 - -/* - * The semaphore operations have a special calling sequence that - * allow us to do a simpler in-line version of them. These routines - * need to convert that sequence back into the C sequence when - * there is contention on the semaphore. - * - * %eax contains the semaphore pointer on entry. Save the C-clobbered - * registers (%eax, %edx and %ecx) except %eax which is either a return - * value or just gets clobbered. Same is true for %edx so make sure GCC - * reloads it after the slow path, by making it hold a temporary, for - * example see ____down_write(). - */ - -#define save_common_regs \ - pushl %ecx - -#define restore_common_regs \ - popl %ecx - - /* Avoid uglifying the argument copying x86-64 needs to do. */ - .macro movq src, dst - .endm - -#else - -/* - * x86-64 rwsem wrappers - * - * This interfaces the inline asm code to the slow-path - * C routines. We need to save the call-clobbered regs - * that the asm does not mark as clobbered, and move the - * argument from %rax to %rdi. - * - * NOTE! We don't need to save %rax, because the functions - * will always return the semaphore pointer in %rax (which - * is also the input argument to these helpers) - * - * The following can clobber %rdx because the asm clobbers it: - * call_rwsem_down_write_failed - * call_rwsem_wake - * but %rdi, %rsi, %rcx, %r8-r11 always need saving. - */ - -#define save_common_regs \ - pushq %rdi; \ - pushq %rsi; \ - pushq %rcx; \ - pushq %r8; \ - pushq %r9; \ - pushq %r10; \ - pushq %r11 - -#define restore_common_regs \ - popq %r11; \ - popq %r10; \ - popq %r9; \ - popq %r8; \ - popq %rcx; \ - popq %rsi; \ - popq %rdi - -#endif - -/* Fix up special calling conventions */ -ENTRY(call_rwsem_down_read_failed) - FRAME_BEGIN - save_common_regs - __ASM_SIZE(push,) %__ASM_REG(dx) - movq %rax,%rdi - call rwsem_down_read_failed - __ASM_SIZE(pop,) %__ASM_REG(dx) - restore_common_regs - FRAME_END - ret -ENDPROC(call_rwsem_down_read_failed) - -ENTRY(call_rwsem_down_read_failed_killable) - FRAME_BEGIN - save_common_regs - __ASM_SIZE(push,) %__ASM_REG(dx) - movq %rax,%rdi - call rwsem_down_read_failed_killable - __ASM_SIZE(pop,) %__ASM_REG(dx) - restore_common_regs - FRAME_END - ret -ENDPROC(call_rwsem_down_read_failed_killable) - -ENTRY(call_rwsem_down_write_failed) - FRAME_BEGIN - save_common_regs - movq %rax,%rdi - call rwsem_down_write_failed - restore_common_regs - FRAME_END - ret -ENDPROC(call_rwsem_down_write_failed) - -ENTRY(call_rwsem_down_write_failed_killable) - FRAME_BEGIN - save_common_regs - movq %rax,%rdi - call rwsem_down_write_failed_killable - restore_common_regs - FRAME_END - ret -ENDPROC(call_rwsem_down_write_failed_killable) - -ENTRY(call_rwsem_wake) - FRAME_BEGIN - /* do nothing if still outstanding active readers */ - __ASM_HALF_SIZE(dec) %__ASM_HALF_REG(dx) - jnz 1f - save_common_regs - movq %rax,%rdi - call rwsem_wake - restore_common_regs -1: FRAME_END - ret -ENDPROC(call_rwsem_wake) - -ENTRY(call_rwsem_downgrade_wake) - FRAME_BEGIN - save_common_regs - __ASM_SIZE(push,) %__ASM_REG(dx) - movq %rax,%rdi - call rwsem_downgrade_wake - __ASM_SIZE(pop,) %__ASM_REG(dx) - restore_common_regs - FRAME_END - ret -ENDPROC(call_rwsem_downgrade_wake) diff --git a/arch/x86/um/Makefile b/arch/x86/um/Makefile index 2d686ae..22554c5 100644 --- a/arch/x86/um/Makefile +++ b/arch/x86/um/Makefile @@ -21,7 +21,6 @@ obj-y += checksum_32.o syscalls_32.o obj-$(CONFIG_ELF_CORE) += elfcore.o subarch-y = ../lib/string_32.o ../lib/atomic64_32.o ../lib/atomic64_cx8_32.o -subarch-$(CONFIG_RWSEM_XCHGADD_ALGORITHM) += ../lib/rwsem.o else diff --git a/arch/xtensa/include/asm/Kbuild b/arch/xtensa/include/asm/Kbuild index e255683..7026b37 100644 --- a/arch/xtensa/include/asm/Kbuild +++ b/arch/xtensa/include/asm/Kbuild @@ -23,7 +23,6 @@ generic-y += mm-arch-hooks.h generic-y += param.h generic-y += percpu.h generic-y += preempt.h -generic-y += rwsem.h generic-y += sections.h generic-y += topology.h generic-y += trace_clock.h diff --git a/include/asm-generic/rwsem.h b/include/asm-generic/rwsem.h deleted file mode 100644 index 93e67a0..0000000 --- a/include/asm-generic/rwsem.h +++ /dev/null @@ -1,140 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0 */ -#ifndef _ASM_GENERIC_RWSEM_H -#define _ASM_GENERIC_RWSEM_H - -#ifndef _LINUX_RWSEM_H -#error "Please don't include <asm/rwsem.h> directly, use <linux/rwsem.h> instead." -#endif - -#ifdef __KERNEL__ - -/* - * R/W semaphores originally for PPC using the stuff in lib/rwsem.c. - * Adapted largely from include/asm-i386/rwsem.h - * by Paul Mackerras <paulus@samba.org>. - */ - -/* - * the semaphore definition - */ -#ifdef CONFIG_64BIT -# define RWSEM_ACTIVE_MASK 0xffffffffL -#else -# define RWSEM_ACTIVE_MASK 0x0000ffffL -#endif - -#define RWSEM_UNLOCKED_VALUE 0x00000000L -#define RWSEM_ACTIVE_BIAS 0x00000001L -#define RWSEM_WAITING_BIAS (-RWSEM_ACTIVE_MASK-1) -#define RWSEM_ACTIVE_READ_BIAS RWSEM_ACTIVE_BIAS -#define RWSEM_ACTIVE_WRITE_BIAS (RWSEM_WAITING_BIAS + RWSEM_ACTIVE_BIAS) - -/* - * lock for reading - */ -static inline void __down_read(struct rw_semaphore *sem) -{ - if (unlikely(atomic_long_inc_return_acquire(&sem->count) <= 0)) - rwsem_down_read_failed(sem); -} - -static inline int __down_read_killable(struct rw_semaphore *sem) -{ - if (unlikely(atomic_long_inc_return_acquire(&sem->count) <= 0)) { - if (IS_ERR(rwsem_down_read_failed_killable(sem))) - return -EINTR; - } - - return 0; -} - -static inline int __down_read_trylock(struct rw_semaphore *sem) -{ - long tmp; - - while ((tmp = atomic_long_read(&sem->count)) >= 0) { - if (tmp == atomic_long_cmpxchg_acquire(&sem->count, tmp, - tmp + RWSEM_ACTIVE_READ_BIAS)) { - return 1; - } - } - return 0; -} - -/* - * lock for writing - */ -static inline void __down_write(struct rw_semaphore *sem) -{ - long tmp; - - tmp = atomic_long_add_return_acquire(RWSEM_ACTIVE_WRITE_BIAS, - &sem->count); - if (unlikely(tmp != RWSEM_ACTIVE_WRITE_BIAS)) - rwsem_down_write_failed(sem); -} - -static inline int __down_write_killable(struct rw_semaphore *sem) -{ - long tmp; - - tmp = atomic_long_add_return_acquire(RWSEM_ACTIVE_WRITE_BIAS, - &sem->count); - if (unlikely(tmp != RWSEM_ACTIVE_WRITE_BIAS)) - if (IS_ERR(rwsem_down_write_failed_killable(sem))) - return -EINTR; - return 0; -} - -static inline int __down_write_trylock(struct rw_semaphore *sem) -{ - long tmp; - - tmp = atomic_long_cmpxchg_acquire(&sem->count, RWSEM_UNLOCKED_VALUE, - RWSEM_ACTIVE_WRITE_BIAS); - return tmp == RWSEM_UNLOCKED_VALUE; -} - -/* - * unlock after reading - */ -static inline void __up_read(struct rw_semaphore *sem) -{ - long tmp; - - tmp = atomic_long_dec_return_release(&sem->count); - if (unlikely(tmp < -1 && (tmp & RWSEM_ACTIVE_MASK) == 0)) - rwsem_wake(sem); -} - -/* - * unlock after writing - */ -static inline void __up_write(struct rw_semaphore *sem) -{ - if (unlikely(atomic_long_sub_return_release(RWSEM_ACTIVE_WRITE_BIAS, - &sem->count) < 0)) - rwsem_wake(sem); -} - -/* - * downgrade write lock to read lock - */ -static inline void __downgrade_write(struct rw_semaphore *sem) -{ - long tmp; - - /* - * When downgrading from exclusive to shared ownership, - * anything inside the write-locked region cannot leak - * into the read side. In contrast, anything in the - * read-locked region is ok to be re-ordered into the - * write side. As such, rely on RELEASE semantics. - */ - tmp = atomic_long_add_return_release(-RWSEM_WAITING_BIAS, &sem->count); - if (tmp < 0) - rwsem_downgrade_wake(sem); -} - -#endif /* __KERNEL__ */ -#endif /* _ASM_GENERIC_RWSEM_H */ diff --git a/include/linux/rwsem.h b/include/linux/rwsem.h index 67dbb57..6e56006 100644 --- a/include/linux/rwsem.h +++ b/include/linux/rwsem.h @@ -57,15 +57,13 @@ struct rw_semaphore { extern struct rw_semaphore *rwsem_wake(struct rw_semaphore *); extern struct rw_semaphore *rwsem_downgrade_wake(struct rw_semaphore *sem); -/* Include the arch specific part */ -#include <asm/rwsem.h> - /* In all implementations count != 0 means locked */ static inline int rwsem_is_locked(struct rw_semaphore *sem) { return atomic_long_read(&sem->count) != 0; } +#define RWSEM_UNLOCKED_VALUE 0L #define __RWSEM_INIT_COUNT(name) .count = ATOMIC_LONG_INIT(RWSEM_UNLOCKED_VALUE) #endif diff --git a/kernel/locking/percpu-rwsem.c b/kernel/locking/percpu-rwsem.c index 883cf1b..f17dad9 100644 --- a/kernel/locking/percpu-rwsem.c +++ b/kernel/locking/percpu-rwsem.c @@ -7,6 +7,8 @@ #include <linux/sched.h> #include <linux/errno.h> +#include "rwsem.h" + int __percpu_init_rwsem(struct percpu_rw_semaphore *sem, const char *name, struct lock_class_key *rwsem_key) { diff --git a/kernel/locking/rwsem.h b/kernel/locking/rwsem.h index bad2bca..067e265 100644 --- a/kernel/locking/rwsem.h +++ b/kernel/locking/rwsem.h @@ -32,6 +32,26 @@ # define DEBUG_RWSEMS_WARN_ON(c) #endif +/* + * R/W semaphores originally for PPC using the stuff in lib/rwsem.c. + * Adapted largely from include/asm-i386/rwsem.h + * by Paul Mackerras <paulus@samba.org>. + */ + +/* + * the semaphore definition + */ +#ifdef CONFIG_64BIT +# define RWSEM_ACTIVE_MASK 0xffffffffL +#else +# define RWSEM_ACTIVE_MASK 0x0000ffffL +#endif + +#define RWSEM_ACTIVE_BIAS 0x00000001L +#define RWSEM_WAITING_BIAS (-RWSEM_ACTIVE_MASK-1) +#define RWSEM_ACTIVE_READ_BIAS RWSEM_ACTIVE_BIAS +#define RWSEM_ACTIVE_WRITE_BIAS (RWSEM_WAITING_BIAS + RWSEM_ACTIVE_BIAS) + #ifdef CONFIG_RWSEM_SPIN_ON_OWNER /* * All writes to owner are protected by WRITE_ONCE() to make sure that @@ -132,3 +152,113 @@ static inline void rwsem_clear_reader_owned(struct rw_semaphore *sem) { } #endif + +#ifdef CONFIG_RWSEM_XCHGADD_ALGORITHM +/* + * lock for reading + */ +static inline void __down_read(struct rw_semaphore *sem) +{ + if (unlikely(atomic_long_inc_return_acquire(&sem->count) <= 0)) + rwsem_down_read_failed(sem); +} + +static inline int __down_read_killable(struct rw_semaphore *sem) +{ + if (unlikely(atomic_long_inc_return_acquire(&sem->count) <= 0)) { + if (IS_ERR(rwsem_down_read_failed_killable(sem))) + return -EINTR; + } + + return 0; +} + +static inline int __down_read_trylock(struct rw_semaphore *sem) +{ + long tmp; + + while ((tmp = atomic_long_read(&sem->count)) >= 0) { + if (tmp == atomic_long_cmpxchg_acquire(&sem->count, tmp, + tmp + RWSEM_ACTIVE_READ_BIAS)) { + return 1; + } + } + return 0; +} + +/* + * lock for writing + */ +static inline void __down_write(struct rw_semaphore *sem) +{ + long tmp; + + tmp = atomic_long_add_return_acquire(RWSEM_ACTIVE_WRITE_BIAS, + &sem->count); + if (unlikely(tmp != RWSEM_ACTIVE_WRITE_BIAS)) + rwsem_down_write_failed(sem); +} + +static inline int __down_write_killable(struct rw_semaphore *sem) +{ + long tmp; + + tmp = atomic_long_add_return_acquire(RWSEM_ACTIVE_WRITE_BIAS, + &sem->count); + if (unlikely(tmp != RWSEM_ACTIVE_WRITE_BIAS)) + if (IS_ERR(rwsem_down_write_failed_killable(sem))) + return -EINTR; + return 0; +} + +static inline int __down_write_trylock(struct rw_semaphore *sem) +{ + long tmp; + + tmp = atomic_long_cmpxchg_acquire(&sem->count, RWSEM_UNLOCKED_VALUE, + RWSEM_ACTIVE_WRITE_BIAS); + return tmp == RWSEM_UNLOCKED_VALUE; +} + +/* + * unlock after reading + */ +static inline void __up_read(struct rw_semaphore *sem) +{ + long tmp; + + tmp = atomic_long_dec_return_release(&sem->count); + if (unlikely(tmp < -1 && (tmp & RWSEM_ACTIVE_MASK) == 0)) + rwsem_wake(sem); +} + +/* + * unlock after writing + */ +static inline void __up_write(struct rw_semaphore *sem) +{ + if (unlikely(atomic_long_sub_return_release(RWSEM_ACTIVE_WRITE_BIAS, + &sem->count) < 0)) + rwsem_wake(sem); +} + +/* + * downgrade write lock to read lock + */ +static inline void __downgrade_write(struct rw_semaphore *sem) +{ + long tmp; + + /* + * When downgrading from exclusive to shared ownership, + * anything inside the write-locked region cannot leak + * into the read side. In contrast, anything in the + * read-locked region is ok to be re-ordered into the + * write side. As such, rely on RELEASE semantics. + */ + tmp = atomic_long_add_return_release(-RWSEM_WAITING_BIAS, &sem->count); + if (tmp < 0) + rwsem_downgrade_wake(sem); +} + +#endif /* CONFIG_RWSEM_XCHGADD_ALGORITHM */ -- 1.8.3.1 ^ permalink raw reply related [flat|nested] 15+ messages in thread
* Re: [PATCH v4 1/3] locking/rwsem: Remove arch specific rwsem files 2019-02-13 22:00 ` [PATCH v4 1/3] locking/rwsem: Remove arch specific rwsem files Waiman Long @ 2019-02-20 14:47 ` Will Deacon 0 siblings, 0 replies; 15+ messages in thread From: Will Deacon @ 2019-02-20 14:47 UTC (permalink / raw) To: Waiman Long Cc: linux-ia64, linux-sh, Peter Zijlstra, linux-mips, H. Peter Anvin, sparclinux, linux-riscv, linux-arch, linux-s390, Davidlohr Bueso, linux-c6x-dev, linux-hexagon, x86, Ingo Molnar, uclinux-h8-devel, linux-xtensa, Arnd Bergmann, linux-um, linux-m68k, openrisc, Borislav Petkov, Thomas Gleixner, linux-arm-kernel, Tim Chen, linux-parisc, Linus Torvalds, linux-kernel, linux-alpha On Wed, Feb 13, 2019 at 05:00:15PM -0500, Waiman Long wrote: > As the generic rwsem-xadd code is using the appropriate acquire and > release versions of the atomic operations, the arch specific rwsem.h > files will not be that much faster than the generic code as long as the > atomic functions are properly implemented. So we can remove those arch > specific rwsem.h and stop building asm/rwsem.h to reduce maintenance > effort. > > Currently, only x86, alpha and ia64 have implemented architecture > specific fast paths. I don't have access to alpha and ia64 systems for > testing, but they are legacy systems that are not likely to be updated > to the latest kernel anyway. > > By using a rwsem microbenchmark, the total locking rates on a 4-socket > 56-core 112-thread x86-64 system before and after the patch were as > follows (mixed means equal # of read and write locks): > > Before Patch After Patch > # of Threads wlock rlock mixed wlock rlock mixed > ------------ ----- ----- ----- ----- ----- ----- > 1 29,201 30,143 29,458 28,615 30,172 29,201 > 2 6,807 13,299 1,171 7,725 15,025 1,804 > 4 6,504 12,755 1,520 7,127 14,286 1,345 > 8 6,762 13,412 764 6,826 13,652 726 > 16 6,693 15,408 662 6,599 15,938 626 > 32 6,145 15,286 496 5,549 15,487 511 > 64 5,812 15,495 60 5,858 15,572 60 > > There were some run-to-run variations for the multi-thread tests. For > x86-64, using the generic C code fast path seems to be a little bit > faster than the assembly version with low lock contention. Looking at > the assembly version of the fast paths, there are assembly to/from C > code wrappers that save and restore all the callee-clobbered registers > (7 registers on x86-64). The assembly generated from the generic C > code doesn't need to do that. That may explain the slight performance > gain here. > > The generic asm rwsem.h can also be merged into kernel/locking/rwsem.h > with no code change as no other code other than those under > kernel/locking needs to access the internal rwsem macros and functions. > > Signed-off-by: Waiman Long <longman@redhat.com> > --- > MAINTAINERS | 1 - > arch/alpha/include/asm/rwsem.h | 211 ----------------------------------- > arch/arm/include/asm/Kbuild | 1 - > arch/arm64/include/asm/Kbuild | 1 - > arch/hexagon/include/asm/Kbuild | 1 - > arch/ia64/include/asm/rwsem.h | 172 ----------------------------- > arch/powerpc/include/asm/Kbuild | 1 - > arch/s390/include/asm/Kbuild | 1 - > arch/sh/include/asm/Kbuild | 1 - > arch/sparc/include/asm/Kbuild | 1 - > arch/x86/include/asm/rwsem.h | 237 ---------------------------------------- > arch/x86/lib/Makefile | 1 - > arch/x86/lib/rwsem.S | 156 -------------------------- > arch/x86/um/Makefile | 1 - > arch/xtensa/include/asm/Kbuild | 1 - > include/asm-generic/rwsem.h | 140 ------------------------ > include/linux/rwsem.h | 4 +- > kernel/locking/percpu-rwsem.c | 2 + > kernel/locking/rwsem.h | 130 ++++++++++++++++++++++ > 19 files changed, 133 insertions(+), 930 deletions(-) > delete mode 100644 arch/alpha/include/asm/rwsem.h > delete mode 100644 arch/ia64/include/asm/rwsem.h > delete mode 100644 arch/x86/include/asm/rwsem.h > delete mode 100644 arch/x86/lib/rwsem.S > delete mode 100644 include/asm-generic/rwsem.h Looks like a nice cleanup, thanks: Acked-by: Will Deacon <will.deacon@arm.com> Will ^ permalink raw reply [flat|nested] 15+ messages in thread
[parent not found: <20190213220016.qVZR7YN6b7PNtJaS2f6lV0L261li-233A9oZ44nZLNQ@z>]
* [PATCH v4 2/3] locking/rwsem: Remove rwsem-spinlock.c & use rwsem-xadd.c for all archs [not found] ` <20190213220016.qVZR7YN6b7PNtJaS2f6lV0L261li-233A9oZ44nZLNQ@z> @ 2019-02-13 22:00 ` Waiman Long [not found] ` <20190214105447.zASTVFPe_Ua4KXhT97AbvJ17RcoLI1qz6hK2MSA1RMk@z> 2019-02-20 14:47 ` Will Deacon 0 siblings, 2 replies; 15+ messages in thread From: Waiman Long @ 2019-02-13 22:00 UTC (permalink / raw) To: Peter Zijlstra, Ingo Molnar, Will Deacon, Thomas Gleixner Cc: linux-kernel, linux-alpha, linux-arm-kernel, linux-c6x-dev, uclinux-h8-devel, linux-hexagon, linux-ia64, linux-m68k, linux-mips, nios2-dev, openrisc, linux-parisc, linuxppc-dev, linux-riscv, linux-s390, linux-sh, sparclinux, linux-um, linux-xtensa, linux-arch, x86, Arnd Bergmann, Borislav Petkov, H. Peter Anvin, Davidlohr Bueso, Linus Torvalds, Andrew Morton, Tim Chen, Waiman Long Currently, we have two different implementation of rwsem: 1) CONFIG_RWSEM_GENERIC_SPINLOCK (rwsem-spinlock.c) 2) CONFIG_RWSEM_XCHGADD_ALGORITHM (rwsem-xadd.c) As we are going to use a single generic implementation for rwsem-xadd.c and no architecture-specific code will be needed, there is no point in keeping two different implementations of rwsem. In most cases, the performance of rwsem-spinlock.c will be worse. It also doesn't get all the performance tuning and optimizations that had been implemented in rwsem-xadd.c over the years. For simplication, we are going to remove rwsem-spinlock.c and make all architectures use a single implementation of rwsem - rwsem-xadd.c. All references to RWSEM_GENERIC_SPINLOCK and RWSEM_XCHGADD_ALGORITHM in the code are removed. Suggested-by: Peter Zijlstra <peterz@infradead.org> Signed-off-by: Waiman Long <longman@redhat.com> --- arch/alpha/Kconfig | 7 - arch/arc/Kconfig | 3 - arch/arm/Kconfig | 4 - arch/arm64/Kconfig | 3 - arch/c6x/Kconfig | 3 - arch/csky/Kconfig | 3 - arch/h8300/Kconfig | 3 - arch/hexagon/Kconfig | 6 - arch/ia64/Kconfig | 4 - arch/m68k/Kconfig | 7 - arch/microblaze/Kconfig | 6 - arch/mips/Kconfig | 7 - arch/nds32/Kconfig | 3 - arch/nios2/Kconfig | 3 - arch/openrisc/Kconfig | 6 - arch/parisc/Kconfig | 6 - arch/powerpc/Kconfig | 7 - arch/riscv/Kconfig | 3 - arch/s390/Kconfig | 6 - arch/sh/Kconfig | 6 - arch/sparc/Kconfig | 8 - arch/unicore32/Kconfig | 6 - arch/x86/Kconfig | 3 - arch/x86/um/Kconfig | 6 - arch/xtensa/Kconfig | 3 - include/linux/rwsem-spinlock.h | 47 ------ include/linux/rwsem.h | 5 - kernel/Kconfig.locks | 2 +- kernel/locking/Makefile | 4 +- kernel/locking/rwsem-spinlock.c | 339 ---------------------------------------- kernel/locking/rwsem.h | 3 - 31 files changed, 2 insertions(+), 520 deletions(-) delete mode 100644 include/linux/rwsem-spinlock.h delete mode 100644 kernel/locking/rwsem-spinlock.c diff --git a/arch/alpha/Kconfig b/arch/alpha/Kconfig index 584a6e1..27c8712 100644 --- a/arch/alpha/Kconfig +++ b/arch/alpha/Kconfig @@ -49,13 +49,6 @@ config MMU bool default y -config RWSEM_GENERIC_SPINLOCK - bool - -config RWSEM_XCHGADD_ALGORITHM - bool - default y - config ARCH_HAS_ILOG2_U32 bool default n diff --git a/arch/arc/Kconfig b/arch/arc/Kconfig index 376366a..c0dd229 100644 --- a/arch/arc/Kconfig +++ b/arch/arc/Kconfig @@ -63,9 +63,6 @@ config SCHED_OMIT_FRAME_POINTER config GENERIC_CSUM def_bool y -config RWSEM_GENERIC_SPINLOCK - def_bool y - config ARCH_DISCONTIGMEM_ENABLE def_bool n diff --git a/arch/arm/Kconfig b/arch/arm/Kconfig index 664e918..4d81f69 100644 --- a/arch/arm/Kconfig +++ b/arch/arm/Kconfig @@ -176,10 +176,6 @@ config TRACE_IRQFLAGS_SUPPORT bool default !CPU_V7M -config RWSEM_XCHGADD_ALGORITHM - bool - default y - config ARCH_HAS_ILOG2_U32 bool diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index a4168d3..24bbcfa 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -238,9 +238,6 @@ config LOCKDEP_SUPPORT config TRACE_IRQFLAGS_SUPPORT def_bool y -config RWSEM_XCHGADD_ALGORITHM - def_bool y - config GENERIC_BUG def_bool y depends on BUG diff --git a/arch/c6x/Kconfig b/arch/c6x/Kconfig index 456e154..f114655 100644 --- a/arch/c6x/Kconfig +++ b/arch/c6x/Kconfig @@ -26,9 +26,6 @@ config MMU config FPU def_bool n -config RWSEM_GENERIC_SPINLOCK - def_bool y - config GENERIC_CALIBRATE_DELAY def_bool y diff --git a/arch/csky/Kconfig b/arch/csky/Kconfig index 398113c..90279a1 100644 --- a/arch/csky/Kconfig +++ b/arch/csky/Kconfig @@ -93,9 +93,6 @@ config GENERIC_HWEIGHT config MMU def_bool y -config RWSEM_GENERIC_SPINLOCK - def_bool y - config STACKTRACE_SUPPORT def_bool y diff --git a/arch/h8300/Kconfig b/arch/h8300/Kconfig index 6472a06..ba33326 100644 --- a/arch/h8300/Kconfig +++ b/arch/h8300/Kconfig @@ -26,9 +26,6 @@ config H8300 config CPU_BIG_ENDIAN def_bool y -config RWSEM_GENERIC_SPINLOCK - def_bool y - config GENERIC_HWEIGHT def_bool y diff --git a/arch/hexagon/Kconfig b/arch/hexagon/Kconfig index fb2fbfc..5d07c8d 100644 --- a/arch/hexagon/Kconfig +++ b/arch/hexagon/Kconfig @@ -64,12 +64,6 @@ config GENERIC_CSUM config GENERIC_IRQ_PROBE def_bool y -config RWSEM_GENERIC_SPINLOCK - def_bool n - -config RWSEM_XCHGADD_ALGORITHM - def_bool y - config GENERIC_HWEIGHT def_bool y diff --git a/arch/ia64/Kconfig b/arch/ia64/Kconfig index 8d7396b..73a26f0 100644 --- a/arch/ia64/Kconfig +++ b/arch/ia64/Kconfig @@ -83,10 +83,6 @@ config STACKTRACE_SUPPORT config GENERIC_LOCKBREAK def_bool n -config RWSEM_XCHGADD_ALGORITHM - bool - default y - config HUGETLB_PAGE_SIZE_VARIABLE bool depends on HUGETLB_PAGE diff --git a/arch/m68k/Kconfig b/arch/m68k/Kconfig index e173ea2..6ace0b2 100644 --- a/arch/m68k/Kconfig +++ b/arch/m68k/Kconfig @@ -31,13 +31,6 @@ config M68K config CPU_BIG_ENDIAN def_bool y -config RWSEM_GENERIC_SPINLOCK - bool - default y - -config RWSEM_XCHGADD_ALGORITHM - bool - config ARCH_HAS_ILOG2_U32 bool diff --git a/arch/microblaze/Kconfig b/arch/microblaze/Kconfig index 58aff26..d459c70 100644 --- a/arch/microblaze/Kconfig +++ b/arch/microblaze/Kconfig @@ -57,15 +57,9 @@ config CPU_LITTLE_ENDIAN endchoice -config RWSEM_GENERIC_SPINLOCK - def_bool y - config ZONE_DMA def_bool y -config RWSEM_XCHGADD_ALGORITHM - bool - config ARCH_HAS_ILOG2_U32 def_bool n diff --git a/arch/mips/Kconfig b/arch/mips/Kconfig index a84c24d..6ad1882 100644 --- a/arch/mips/Kconfig +++ b/arch/mips/Kconfig @@ -1037,13 +1037,6 @@ source "arch/mips/paravirt/Kconfig" endmenu -config RWSEM_GENERIC_SPINLOCK - bool - default y - -config RWSEM_XCHGADD_ALGORITHM - bool - config GENERIC_HWEIGHT bool default y diff --git a/arch/nds32/Kconfig b/arch/nds32/Kconfig index dda1906..6a3c53d 100644 --- a/arch/nds32/Kconfig +++ b/arch/nds32/Kconfig @@ -59,9 +59,6 @@ config GENERIC_LOCKBREAK def_bool y depends on PREEMPT -config RWSEM_GENERIC_SPINLOCK - def_bool y - config TRACE_IRQFLAGS_SUPPORT def_bool y diff --git a/arch/nios2/Kconfig b/arch/nios2/Kconfig index 532343e..715e6c0 100644 --- a/arch/nios2/Kconfig +++ b/arch/nios2/Kconfig @@ -39,9 +39,6 @@ config NO_IOPORT_MAP config FPU def_bool n -config RWSEM_GENERIC_SPINLOCK - def_bool y - config TRACE_IRQFLAGS_SUPPORT def_bool n diff --git a/arch/openrisc/Kconfig b/arch/openrisc/Kconfig index 09ab59e..9f51da5 100644 --- a/arch/openrisc/Kconfig +++ b/arch/openrisc/Kconfig @@ -42,12 +42,6 @@ config CPU_BIG_ENDIAN config MMU def_bool y -config RWSEM_GENERIC_SPINLOCK - def_bool y - -config RWSEM_XCHGADD_ALGORITHM - def_bool n - config GENERIC_HWEIGHT def_bool y diff --git a/arch/parisc/Kconfig b/arch/parisc/Kconfig index 7ca2c3e..8efbeea 100644 --- a/arch/parisc/Kconfig +++ b/arch/parisc/Kconfig @@ -74,12 +74,6 @@ config GENERIC_LOCKBREAK default y depends on SMP && PREEMPT -config RWSEM_GENERIC_SPINLOCK - def_bool y - -config RWSEM_XCHGADD_ALGORITHM - bool - config ARCH_HAS_ILOG2_U32 bool default n diff --git a/arch/powerpc/Kconfig b/arch/powerpc/Kconfig index 2890d36..466e90f 100644 --- a/arch/powerpc/Kconfig +++ b/arch/powerpc/Kconfig @@ -103,13 +103,6 @@ config LOCKDEP_SUPPORT bool default y -config RWSEM_GENERIC_SPINLOCK - bool - -config RWSEM_XCHGADD_ALGORITHM - bool - default y - config GENERIC_LOCKBREAK bool default y diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index 515fc3c..2b34482 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -69,9 +69,6 @@ config STACKTRACE_SUPPORT config TRACE_IRQFLAGS_SUPPORT def_bool y -config RWSEM_GENERIC_SPINLOCK - def_bool y - config GENERIC_BUG def_bool y depends on BUG diff --git a/arch/s390/Kconfig b/arch/s390/Kconfig index ed554b0..0ec9d8e 100644 --- a/arch/s390/Kconfig +++ b/arch/s390/Kconfig @@ -14,12 +14,6 @@ config LOCKDEP_SUPPORT config STACKTRACE_SUPPORT def_bool y -config RWSEM_GENERIC_SPINLOCK - bool - -config RWSEM_XCHGADD_ALGORITHM - def_bool y - config ARCH_HAS_ILOG2_U32 def_bool n diff --git a/arch/sh/Kconfig b/arch/sh/Kconfig index a9c36f9..4f8c747 100644 --- a/arch/sh/Kconfig +++ b/arch/sh/Kconfig @@ -89,12 +89,6 @@ config ARCH_DEFCONFIG default "arch/sh/configs/shx3_defconfig" if SUPERH32 default "arch/sh/configs/cayman_defconfig" if SUPERH64 -config RWSEM_GENERIC_SPINLOCK - def_bool y - -config RWSEM_XCHGADD_ALGORITHM - bool - config GENERIC_BUG def_bool y depends on BUG && SUPERH32 diff --git a/arch/sparc/Kconfig b/arch/sparc/Kconfig index d5dd652..6d0036a 100644 --- a/arch/sparc/Kconfig +++ b/arch/sparc/Kconfig @@ -190,14 +190,6 @@ config NR_CPUS source "kernel/Kconfig.hz" -config RWSEM_GENERIC_SPINLOCK - bool - default y if SPARC32 - -config RWSEM_XCHGADD_ALGORITHM - bool - default y if SPARC64 - config GENERIC_HWEIGHT bool default y diff --git a/arch/unicore32/Kconfig b/arch/unicore32/Kconfig index c3a41bf..e080087 100644 --- a/arch/unicore32/Kconfig +++ b/arch/unicore32/Kconfig @@ -38,12 +38,6 @@ config STACKTRACE_SUPPORT config LOCKDEP_SUPPORT def_bool y -config RWSEM_GENERIC_SPINLOCK - def_bool y - -config RWSEM_XCHGADD_ALGORITHM - bool - config ARCH_HAS_ILOG2_U32 bool diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index 6826143..be6cc2c 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -269,9 +269,6 @@ config ARCH_MAY_HAVE_PC_FDC def_bool y depends on ISA_DMA_API -config RWSEM_XCHGADD_ALGORITHM - def_bool y - config GENERIC_CALIBRATE_DELAY def_bool y diff --git a/arch/x86/um/Kconfig b/arch/x86/um/Kconfig index f518b47..4dc3217 100644 --- a/arch/x86/um/Kconfig +++ b/arch/x86/um/Kconfig @@ -32,12 +32,6 @@ config ARCH_DEFCONFIG default "arch/um/configs/i386_defconfig" if X86_32 default "arch/um/configs/x86_64_defconfig" if X86_64 -config RWSEM_XCHGADD_ALGORITHM - def_bool 64BIT - -config RWSEM_GENERIC_SPINLOCK - def_bool !RWSEM_XCHGADD_ALGORITHM - config 3_LEVEL_PGTABLES bool "Three-level pagetables" if !64BIT default 64BIT diff --git a/arch/xtensa/Kconfig b/arch/xtensa/Kconfig index ce91682..4f0a903 100644 --- a/arch/xtensa/Kconfig +++ b/arch/xtensa/Kconfig @@ -43,9 +43,6 @@ config XTENSA with reasonable minimum requirements. The Xtensa Linux project has a home page at <http://www.linux-xtensa.org/>. -config RWSEM_XCHGADD_ALGORITHM - def_bool y - config GENERIC_HWEIGHT def_bool y diff --git a/include/linux/rwsem-spinlock.h b/include/linux/rwsem-spinlock.h deleted file mode 100644 index e475683..0000000 --- a/include/linux/rwsem-spinlock.h +++ /dev/null @@ -1,47 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0 */ -/* rwsem-spinlock.h: fallback C implementation - * - * Copyright (c) 2001 David Howells (dhowells@redhat.com). - * - Derived partially from ideas by Andrea Arcangeli <andrea@suse.de> - * - Derived also from comments by Linus - */ - -#ifndef _LINUX_RWSEM_SPINLOCK_H -#define _LINUX_RWSEM_SPINLOCK_H - -#ifndef _LINUX_RWSEM_H -#error "please don't include linux/rwsem-spinlock.h directly, use linux/rwsem.h instead" -#endif - -#ifdef __KERNEL__ -/* - * the rw-semaphore definition - * - if count is 0 then there are no active readers or writers - * - if count is +ve then that is the number of active readers - * - if count is -1 then there is one active writer - * - if wait_list is not empty, then there are processes waiting for the semaphore - */ -struct rw_semaphore { - __s32 count; - raw_spinlock_t wait_lock; - struct list_head wait_list; -#ifdef CONFIG_DEBUG_LOCK_ALLOC - struct lockdep_map dep_map; -#endif -}; - -#define RWSEM_UNLOCKED_VALUE 0x00000000 - -extern void __down_read(struct rw_semaphore *sem); -extern int __must_check __down_read_killable(struct rw_semaphore *sem); -extern int __down_read_trylock(struct rw_semaphore *sem); -extern void __down_write(struct rw_semaphore *sem); -extern int __must_check __down_write_killable(struct rw_semaphore *sem); -extern int __down_write_trylock(struct rw_semaphore *sem); -extern void __up_read(struct rw_semaphore *sem); -extern void __up_write(struct rw_semaphore *sem); -extern void __downgrade_write(struct rw_semaphore *sem); -extern int rwsem_is_locked(struct rw_semaphore *sem); - -#endif /* __KERNEL__ */ -#endif /* _LINUX_RWSEM_SPINLOCK_H */ diff --git a/include/linux/rwsem.h b/include/linux/rwsem.h index 6e56006..0fc4106 100644 --- a/include/linux/rwsem.h +++ b/include/linux/rwsem.h @@ -22,10 +22,6 @@ struct rw_semaphore; -#ifdef CONFIG_RWSEM_GENERIC_SPINLOCK -#include <linux/rwsem-spinlock.h> /* use a generic implementation */ -#define __RWSEM_INIT_COUNT(name) .count = RWSEM_UNLOCKED_VALUE -#else /* All arch specific implementations share the same struct */ struct rw_semaphore { atomic_long_t count; @@ -65,7 +61,6 @@ static inline int rwsem_is_locked(struct rw_semaphore *sem) #define RWSEM_UNLOCKED_VALUE 0L #define __RWSEM_INIT_COUNT(name) .count = ATOMIC_LONG_INIT(RWSEM_UNLOCKED_VALUE) -#endif /* Common initializer macros and functions */ diff --git a/kernel/Kconfig.locks b/kernel/Kconfig.locks index 84d882f..74775ed 100644 --- a/kernel/Kconfig.locks +++ b/kernel/Kconfig.locks @@ -229,7 +229,7 @@ config MUTEX_SPIN_ON_OWNER config RWSEM_SPIN_ON_OWNER def_bool y - depends on SMP && RWSEM_XCHGADD_ALGORITHM && ARCH_SUPPORTS_ATOMIC_RMW + depends on SMP && ARCH_SUPPORTS_ATOMIC_RMW config LOCK_SPIN_ON_OWNER def_bool y diff --git a/kernel/locking/Makefile b/kernel/locking/Makefile index 392c7f2..1af83e9 100644 --- a/kernel/locking/Makefile +++ b/kernel/locking/Makefile @@ -3,7 +3,7 @@ # and is generally not a function of system call inputs. KCOV_INSTRUMENT := n -obj-y += mutex.o semaphore.o rwsem.o percpu-rwsem.o +obj-y += mutex.o semaphore.o rwsem.o percpu-rwsem.o rwsem-xadd.o ifdef CONFIG_FUNCTION_TRACER CFLAGS_REMOVE_lockdep.o = $(CC_FLAGS_FTRACE) @@ -25,8 +25,6 @@ obj-$(CONFIG_RT_MUTEXES) += rtmutex.o obj-$(CONFIG_DEBUG_RT_MUTEXES) += rtmutex-debug.o obj-$(CONFIG_DEBUG_SPINLOCK) += spinlock.o obj-$(CONFIG_DEBUG_SPINLOCK) += spinlock_debug.o -obj-$(CONFIG_RWSEM_GENERIC_SPINLOCK) += rwsem-spinlock.o -obj-$(CONFIG_RWSEM_XCHGADD_ALGORITHM) += rwsem-xadd.o obj-$(CONFIG_QUEUED_RWLOCKS) += qrwlock.o obj-$(CONFIG_LOCK_TORTURE_TEST) += locktorture.o obj-$(CONFIG_WW_MUTEX_SELFTEST) += test-ww_mutex.o diff --git a/kernel/locking/rwsem-spinlock.c b/kernel/locking/rwsem-spinlock.c deleted file mode 100644 index a7ffb2a..0000000 --- a/kernel/locking/rwsem-spinlock.c +++ /dev/null @@ -1,339 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0 -/* rwsem-spinlock.c: R/W semaphores: contention handling functions for - * generic spinlock implementation - * - * Copyright (c) 2001 David Howells (dhowells@redhat.com). - * - Derived partially from idea by Andrea Arcangeli <andrea@suse.de> - * - Derived also from comments by Linus - */ -#include <linux/rwsem.h> -#include <linux/sched/signal.h> -#include <linux/sched/debug.h> -#include <linux/export.h> - -enum rwsem_waiter_type { - RWSEM_WAITING_FOR_WRITE, - RWSEM_WAITING_FOR_READ -}; - -struct rwsem_waiter { - struct list_head list; - struct task_struct *task; - enum rwsem_waiter_type type; -}; - -int rwsem_is_locked(struct rw_semaphore *sem) -{ - int ret = 1; - unsigned long flags; - - if (raw_spin_trylock_irqsave(&sem->wait_lock, flags)) { - ret = (sem->count != 0); - raw_spin_unlock_irqrestore(&sem->wait_lock, flags); - } - return ret; -} -EXPORT_SYMBOL(rwsem_is_locked); - -/* - * initialise the semaphore - */ -void __init_rwsem(struct rw_semaphore *sem, const char *name, - struct lock_class_key *key) -{ -#ifdef CONFIG_DEBUG_LOCK_ALLOC - /* - * Make sure we are not reinitializing a held semaphore: - */ - debug_check_no_locks_freed((void *)sem, sizeof(*sem)); - lockdep_init_map(&sem->dep_map, name, key, 0); -#endif - sem->count = 0; - raw_spin_lock_init(&sem->wait_lock); - INIT_LIST_HEAD(&sem->wait_list); -} -EXPORT_SYMBOL(__init_rwsem); - -/* - * handle the lock release when processes blocked on it that can now run - * - if we come here, then: - * - the 'active count' _reached_ zero - * - the 'waiting count' is non-zero - * - the spinlock must be held by the caller - * - woken process blocks are discarded from the list after having task zeroed - * - writers are only woken if wakewrite is non-zero - */ -static inline struct rw_semaphore * -__rwsem_do_wake(struct rw_semaphore *sem, int wakewrite) -{ - struct rwsem_waiter *waiter; - struct task_struct *tsk; - int woken; - - waiter = list_entry(sem->wait_list.next, struct rwsem_waiter, list); - - if (waiter->type == RWSEM_WAITING_FOR_WRITE) { - if (wakewrite) - /* Wake up a writer. Note that we do not grant it the - * lock - it will have to acquire it when it runs. */ - wake_up_process(waiter->task); - goto out; - } - - /* grant an infinite number of read locks to the front of the queue */ - woken = 0; - do { - struct list_head *next = waiter->list.next; - - list_del(&waiter->list); - tsk = waiter->task; - /* - * Make sure we do not wakeup the next reader before - * setting the nil condition to grant the next reader; - * otherwise we could miss the wakeup on the other - * side and end up sleeping again. See the pairing - * in rwsem_down_read_failed(). - */ - smp_mb(); - waiter->task = NULL; - wake_up_process(tsk); - put_task_struct(tsk); - woken++; - if (next == &sem->wait_list) - break; - waiter = list_entry(next, struct rwsem_waiter, list); - } while (waiter->type != RWSEM_WAITING_FOR_WRITE); - - sem->count += woken; - - out: - return sem; -} - -/* - * wake a single writer - */ -static inline struct rw_semaphore * -__rwsem_wake_one_writer(struct rw_semaphore *sem) -{ - struct rwsem_waiter *waiter; - - waiter = list_entry(sem->wait_list.next, struct rwsem_waiter, list); - wake_up_process(waiter->task); - - return sem; -} - -/* - * get a read lock on the semaphore - */ -int __sched __down_read_common(struct rw_semaphore *sem, int state) -{ - struct rwsem_waiter waiter; - unsigned long flags; - - raw_spin_lock_irqsave(&sem->wait_lock, flags); - - if (sem->count >= 0 && list_empty(&sem->wait_list)) { - /* granted */ - sem->count++; - raw_spin_unlock_irqrestore(&sem->wait_lock, flags); - goto out; - } - - /* set up my own style of waitqueue */ - waiter.task = current; - waiter.type = RWSEM_WAITING_FOR_READ; - get_task_struct(current); - - list_add_tail(&waiter.list, &sem->wait_list); - - /* wait to be given the lock */ - for (;;) { - if (!waiter.task) - break; - if (signal_pending_state(state, current)) - goto out_nolock; - set_current_state(state); - raw_spin_unlock_irqrestore(&sem->wait_lock, flags); - schedule(); - raw_spin_lock_irqsave(&sem->wait_lock, flags); - } - - raw_spin_unlock_irqrestore(&sem->wait_lock, flags); - out: - return 0; - -out_nolock: - /* - * We didn't take the lock, so that there is a writer, which - * is owner or the first waiter of the sem. If it's a waiter, - * it will be woken by current owner. Not need to wake anybody. - */ - list_del(&waiter.list); - raw_spin_unlock_irqrestore(&sem->wait_lock, flags); - return -EINTR; -} - -void __sched __down_read(struct rw_semaphore *sem) -{ - __down_read_common(sem, TASK_UNINTERRUPTIBLE); -} - -int __sched __down_read_killable(struct rw_semaphore *sem) -{ - return __down_read_common(sem, TASK_KILLABLE); -} - -/* - * trylock for reading -- returns 1 if successful, 0 if contention - */ -int __down_read_trylock(struct rw_semaphore *sem) -{ - unsigned long flags; - int ret = 0; - - - raw_spin_lock_irqsave(&sem->wait_lock, flags); - - if (sem->count >= 0 && list_empty(&sem->wait_list)) { - /* granted */ - sem->count++; - ret = 1; - } - - raw_spin_unlock_irqrestore(&sem->wait_lock, flags); - - return ret; -} - -/* - * get a write lock on the semaphore - */ -int __sched __down_write_common(struct rw_semaphore *sem, int state) -{ - struct rwsem_waiter waiter; - unsigned long flags; - int ret = 0; - - raw_spin_lock_irqsave(&sem->wait_lock, flags); - - /* set up my own style of waitqueue */ - waiter.task = current; - waiter.type = RWSEM_WAITING_FOR_WRITE; - list_add_tail(&waiter.list, &sem->wait_list); - - /* wait for someone to release the lock */ - for (;;) { - /* - * That is the key to support write lock stealing: allows the - * task already on CPU to get the lock soon rather than put - * itself into sleep and waiting for system woke it or someone - * else in the head of the wait list up. - */ - if (sem->count == 0) - break; - if (signal_pending_state(state, current)) - goto out_nolock; - - set_current_state(state); - raw_spin_unlock_irqrestore(&sem->wait_lock, flags); - schedule(); - raw_spin_lock_irqsave(&sem->wait_lock, flags); - } - /* got the lock */ - sem->count = -1; - list_del(&waiter.list); - - raw_spin_unlock_irqrestore(&sem->wait_lock, flags); - - return ret; - -out_nolock: - list_del(&waiter.list); - if (!list_empty(&sem->wait_list) && sem->count >= 0) - __rwsem_do_wake(sem, 0); - raw_spin_unlock_irqrestore(&sem->wait_lock, flags); - - return -EINTR; -} - -void __sched __down_write(struct rw_semaphore *sem) -{ - __down_write_common(sem, TASK_UNINTERRUPTIBLE); -} - -int __sched __down_write_killable(struct rw_semaphore *sem) -{ - return __down_write_common(sem, TASK_KILLABLE); -} - -/* - * trylock for writing -- returns 1 if successful, 0 if contention - */ -int __down_write_trylock(struct rw_semaphore *sem) -{ - unsigned long flags; - int ret = 0; - - raw_spin_lock_irqsave(&sem->wait_lock, flags); - - if (sem->count == 0) { - /* got the lock */ - sem->count = -1; - ret = 1; - } - - raw_spin_unlock_irqrestore(&sem->wait_lock, flags); - - return ret; -} - -/* - * release a read lock on the semaphore - */ -void __up_read(struct rw_semaphore *sem) -{ - unsigned long flags; - - raw_spin_lock_irqsave(&sem->wait_lock, flags); - - if (--sem->count == 0 && !list_empty(&sem->wait_list)) - sem = __rwsem_wake_one_writer(sem); - - raw_spin_unlock_irqrestore(&sem->wait_lock, flags); -} - -/* - * release a write lock on the semaphore - */ -void __up_write(struct rw_semaphore *sem) -{ - unsigned long flags; - - raw_spin_lock_irqsave(&sem->wait_lock, flags); - - sem->count = 0; - if (!list_empty(&sem->wait_list)) - sem = __rwsem_do_wake(sem, 1); - - raw_spin_unlock_irqrestore(&sem->wait_lock, flags); -} - -/* - * downgrade a write lock into a read lock - * - just wake up any readers at the front of the queue - */ -void __downgrade_write(struct rw_semaphore *sem) -{ - unsigned long flags; - - raw_spin_lock_irqsave(&sem->wait_lock, flags); - - sem->count = 1; - if (!list_empty(&sem->wait_list)) - sem = __rwsem_do_wake(sem, 0); - - raw_spin_unlock_irqrestore(&sem->wait_lock, flags); -} - diff --git a/kernel/locking/rwsem.h b/kernel/locking/rwsem.h index 067e265..45ee002 100644 --- a/kernel/locking/rwsem.h +++ b/kernel/locking/rwsem.h @@ -153,7 +153,6 @@ static inline void rwsem_clear_reader_owned(struct rw_semaphore *sem) } #endif -#ifdef CONFIG_RWSEM_XCHGADD_ALGORITHM /* * lock for reading */ @@ -260,5 +259,3 @@ static inline void __downgrade_write(struct rw_semaphore *sem) if (tmp < 0) rwsem_downgrade_wake(sem); } - -#endif /* CONFIG_RWSEM_XCHGADD_ALGORITHM */ -- 1.8.3.1 ^ permalink raw reply related [flat|nested] 15+ messages in thread
[parent not found: <20190214105447.zASTVFPe_Ua4KXhT97AbvJ17RcoLI1qz6hK2MSA1RMk@z>]
* Re: [PATCH v4 2/3] locking/rwsem: Remove rwsem-spinlock.c & use rwsem-xadd.c for all archs [not found] ` <20190214105447.zASTVFPe_Ua4KXhT97AbvJ17RcoLI1qz6hK2MSA1RMk@z> @ 2019-02-14 10:54 ` Geert Uytterhoeven [not found] ` <20190214111211.Zz4JLUBLm4iMV2I1EaQtWszvprA6uZ_UN5mNKMeBl04@z> 0 siblings, 1 reply; 15+ messages in thread From: Geert Uytterhoeven @ 2019-02-14 10:54 UTC (permalink / raw) To: Waiman Long, Christoph Hellwig Cc: Peter Zijlstra, Ingo Molnar, Will Deacon, Thomas Gleixner, linux-ia64@vger.kernel.org, Linux-sh list, linux-mips, H. Peter Anvin, sparclinux, linux-riscv, Linux-Arch, linux-s390, Davidlohr Bueso, linux-c6x-dev, open list:QUALCOMM HEXAGON..., the arch/x86 maintainers, moderated list:H8/300 ARCHITECTURE, linux-xtensa, Arnd Bergmann, linux-um, linux-m68k, Openrisc, Borislav Petkov, Linux ARM, Tim Chen, Parisc List, Linus Torvalds, Linux Kernel Mailing List, alpha, nios2-dev, Andrew Morton, linuxppc-dev On Wed, Feb 13, 2019 at 11:01 PM Waiman Long <longman@redhat.com> wrote: > Currently, we have two different implementation of rwsem: > 1) CONFIG_RWSEM_GENERIC_SPINLOCK (rwsem-spinlock.c) > 2) CONFIG_RWSEM_XCHGADD_ALGORITHM (rwsem-xadd.c) > > As we are going to use a single generic implementation for rwsem-xadd.c > and no architecture-specific code will be needed, there is no point > in keeping two different implementations of rwsem. In most cases, the > performance of rwsem-spinlock.c will be worse. It also doesn't get all > the performance tuning and optimizations that had been implemented in > rwsem-xadd.c over the years. > > For simplication, we are going to remove rwsem-spinlock.c and make all > architectures use a single implementation of rwsem - rwsem-xadd.c. > > All references to RWSEM_GENERIC_SPINLOCK and RWSEM_XCHGADD_ALGORITHM > in the code are removed. > > Suggested-by: Peter Zijlstra <peterz@infradead.org> > Signed-off-by: Waiman Long <longman@redhat.com> Note that this conflicts with "[PATCH 03/11] kernel/locks: consolidate RWSEM_GENERIC_* options" https://lore.kernel.org/lkml/20190213174005.28785-4-hch@lst.de/ Gr{oetje,eeting}s, Geert -- Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org In personal conversations with technical people, I call myself a hacker. But when I'm talking to journalists I just say "programmer" or something like that. -- Linus Torvalds ^ permalink raw reply [flat|nested] 15+ messages in thread
[parent not found: <20190214111211.Zz4JLUBLm4iMV2I1EaQtWszvprA6uZ_UN5mNKMeBl04@z>]
* Re: [PATCH v4 2/3] locking/rwsem: Remove rwsem-spinlock.c & use rwsem-xadd.c for all archs [not found] ` <20190214111211.Zz4JLUBLm4iMV2I1EaQtWszvprA6uZ_UN5mNKMeBl04@z> @ 2019-02-14 11:12 ` Peter Zijlstra 0 siblings, 0 replies; 15+ messages in thread From: Peter Zijlstra @ 2019-02-14 11:12 UTC (permalink / raw) To: Geert Uytterhoeven Cc: Waiman Long, Christoph Hellwig, Ingo Molnar, Will Deacon, Thomas Gleixner, linux-ia64@vger.kernel.org, Linux-sh list, linux-mips, H. Peter Anvin, sparclinux, linux-riscv, Linux-Arch, linux-s390, Davidlohr Bueso, linux-c6x-dev, open list:QUALCOMM HEXAGON..., the arch/x86 maintainers, moderated list:H8/300 ARCHITECTURE, linux-xtensa, Arnd Bergmann, linux-um, linux-m68k, Openrisc, Borislav Petkov, Linux ARM, Tim Chen, Parisc List, Linus Torvalds, Linux Kernel Mailing List, alpha, nios2-dev, Andrew Morton, linuxppc-dev On Thu, Feb 14, 2019 at 11:54:47AM +0100, Geert Uytterhoeven wrote: > On Wed, Feb 13, 2019 at 11:01 PM Waiman Long <longman@redhat.com> wrote: > > Currently, we have two different implementation of rwsem: > > 1) CONFIG_RWSEM_GENERIC_SPINLOCK (rwsem-spinlock.c) > > 2) CONFIG_RWSEM_XCHGADD_ALGORITHM (rwsem-xadd.c) > > > > As we are going to use a single generic implementation for rwsem-xadd.c > > and no architecture-specific code will be needed, there is no point > > in keeping two different implementations of rwsem. In most cases, the > > performance of rwsem-spinlock.c will be worse. It also doesn't get all > > the performance tuning and optimizations that had been implemented in > > rwsem-xadd.c over the years. > > > > For simplication, we are going to remove rwsem-spinlock.c and make all > > architectures use a single implementation of rwsem - rwsem-xadd.c. > > > > All references to RWSEM_GENERIC_SPINLOCK and RWSEM_XCHGADD_ALGORITHM > > in the code are removed. > > > > Suggested-by: Peter Zijlstra <peterz@infradead.org> > > Signed-off-by: Waiman Long <longman@redhat.com> > > Note that this conflicts with "[PATCH 03/11] kernel/locks: consolidate > RWSEM_GENERIC_* options" > https://lore.kernel.org/lkml/20190213174005.28785-4-hch@lst.de/ *sigh*.. of that never was Cc'ed to locking people :/ ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH v4 2/3] locking/rwsem: Remove rwsem-spinlock.c & use rwsem-xadd.c for all archs 2019-02-13 22:00 ` [PATCH v4 2/3] locking/rwsem: Remove rwsem-spinlock.c & use rwsem-xadd.c for all archs Waiman Long [not found] ` <20190214105447.zASTVFPe_Ua4KXhT97AbvJ17RcoLI1qz6hK2MSA1RMk@z> @ 2019-02-20 14:47 ` Will Deacon 1 sibling, 0 replies; 15+ messages in thread From: Will Deacon @ 2019-02-20 14:47 UTC (permalink / raw) To: Waiman Long Cc: Peter Zijlstra, Ingo Molnar, Thomas Gleixner, linux-kernel, linux-alpha, linux-arm-kernel, linux-c6x-dev, uclinux-h8-devel, linux-hexagon, linux-ia64, linux-m68k, linux-mips, nios2-dev, openrisc, linux-parisc, linuxppc-dev, linux-riscv, linux-s390, linux-sh, sparclinux, linux-um, linux-xtensa, linux-arch, x86, Arnd Bergmann, Borislav On Wed, Feb 13, 2019 at 05:00:16PM -0500, Waiman Long wrote: > Currently, we have two different implementation of rwsem: > 1) CONFIG_RWSEM_GENERIC_SPINLOCK (rwsem-spinlock.c) > 2) CONFIG_RWSEM_XCHGADD_ALGORITHM (rwsem-xadd.c) > > As we are going to use a single generic implementation for rwsem-xadd.c > and no architecture-specific code will be needed, there is no point > in keeping two different implementations of rwsem. In most cases, the > performance of rwsem-spinlock.c will be worse. It also doesn't get all > the performance tuning and optimizations that had been implemented in > rwsem-xadd.c over the years. > > For simplication, we are going to remove rwsem-spinlock.c and make all > architectures use a single implementation of rwsem - rwsem-xadd.c. > > All references to RWSEM_GENERIC_SPINLOCK and RWSEM_XCHGADD_ALGORITHM > in the code are removed. > > Suggested-by: Peter Zijlstra <peterz@infradead.org> > Signed-off-by: Waiman Long <longman@redhat.com> > --- > arch/alpha/Kconfig | 7 - > arch/arc/Kconfig | 3 - > arch/arm/Kconfig | 4 - > arch/arm64/Kconfig | 3 - > arch/c6x/Kconfig | 3 - > arch/csky/Kconfig | 3 - > arch/h8300/Kconfig | 3 - > arch/hexagon/Kconfig | 6 - > arch/ia64/Kconfig | 4 - > arch/m68k/Kconfig | 7 - > arch/microblaze/Kconfig | 6 - > arch/mips/Kconfig | 7 - > arch/nds32/Kconfig | 3 - > arch/nios2/Kconfig | 3 - > arch/openrisc/Kconfig | 6 - > arch/parisc/Kconfig | 6 - > arch/powerpc/Kconfig | 7 - > arch/riscv/Kconfig | 3 - > arch/s390/Kconfig | 6 - > arch/sh/Kconfig | 6 - > arch/sparc/Kconfig | 8 - > arch/unicore32/Kconfig | 6 - > arch/x86/Kconfig | 3 - > arch/x86/um/Kconfig | 6 - > arch/xtensa/Kconfig | 3 - > include/linux/rwsem-spinlock.h | 47 ------ > include/linux/rwsem.h | 5 - > kernel/Kconfig.locks | 2 +- > kernel/locking/Makefile | 4 +- > kernel/locking/rwsem-spinlock.c | 339 ---------------------------------------- > kernel/locking/rwsem.h | 3 - > 31 files changed, 2 insertions(+), 520 deletions(-) > delete mode 100644 include/linux/rwsem-spinlock.h > delete mode 100644 kernel/locking/rwsem-spinlock.c Another nice cleanup, and it looks like the optimistic spinning in the xadd implementation is predicated on ARCH_SUPPORTS_ATOMIC_RMW, so we're all good there too. Acked-by: Will Deacon <will.deacon@arm.com> Will ^ permalink raw reply [flat|nested] 15+ messages in thread
[parent not found: <20190213220017.fPHty3eMIFui2_R0HHBvLImq4wVl1Ii36BMjgrdkIxg@z>]
* [PATCH v4 3/3] locking/rwsem: Optimize down_read_trylock() [not found] ` <20190213220017.fPHty3eMIFui2_R0HHBvLImq4wVl1Ii36BMjgrdkIxg@z> @ 2019-02-13 22:00 ` Waiman Long 2019-02-21 14:14 ` Will Deacon 0 siblings, 1 reply; 15+ messages in thread From: Waiman Long @ 2019-02-13 22:00 UTC (permalink / raw) To: Peter Zijlstra, Ingo Molnar, Will Deacon, Thomas Gleixner Cc: linux-kernel, linux-alpha, linux-arm-kernel, linux-c6x-dev, uclinux-h8-devel, linux-hexagon, linux-ia64, linux-m68k, linux-mips, nios2-dev, openrisc, linux-parisc, linuxppc-dev, linux-riscv, linux-s390, linux-sh, sparclinux, linux-um, linux-xtensa, linux-arch, x86, Arnd Bergmann, Borislav Petkov, H. Peter Anvin, Davidlohr Bueso, Linus Torvalds, Andrew Morton, Tim Chen, Waiman Long Modify __down_read_trylock() to optimize for an unlocked rwsem and make it generate slightly better code. Before this patch, down_read_trylock: 0x0000000000000000 <+0>: callq 0x5 <down_read_trylock+5> 0x0000000000000005 <+5>: jmp 0x18 <down_read_trylock+24> 0x0000000000000007 <+7>: lea 0x1(%rdx),%rcx 0x000000000000000b <+11>: mov %rdx,%rax 0x000000000000000e <+14>: lock cmpxchg %rcx,(%rdi) 0x0000000000000013 <+19>: cmp %rax,%rdx 0x0000000000000016 <+22>: je 0x23 <down_read_trylock+35> 0x0000000000000018 <+24>: mov (%rdi),%rdx 0x000000000000001b <+27>: test %rdx,%rdx 0x000000000000001e <+30>: jns 0x7 <down_read_trylock+7> 0x0000000000000020 <+32>: xor %eax,%eax 0x0000000000000022 <+34>: retq 0x0000000000000023 <+35>: mov %gs:0x0,%rax 0x000000000000002c <+44>: or $0x3,%rax 0x0000000000000030 <+48>: mov %rax,0x20(%rdi) 0x0000000000000034 <+52>: mov $0x1,%eax 0x0000000000000039 <+57>: retq After patch, down_read_trylock: 0x0000000000000000 <+0>: callq 0x5 <down_read_trylock+5> 0x0000000000000005 <+5>: xor %eax,%eax 0x0000000000000007 <+7>: lea 0x1(%rax),%rdx 0x000000000000000b <+11>: lock cmpxchg %rdx,(%rdi) 0x0000000000000010 <+16>: jne 0x29 <down_read_trylock+41> 0x0000000000000012 <+18>: mov %gs:0x0,%rax 0x000000000000001b <+27>: or $0x3,%rax 0x000000000000001f <+31>: mov %rax,0x20(%rdi) 0x0000000000000023 <+35>: mov $0x1,%eax 0x0000000000000028 <+40>: retq 0x0000000000000029 <+41>: test %rax,%rax 0x000000000000002c <+44>: jns 0x7 <down_read_trylock+7> 0x000000000000002e <+46>: xor %eax,%eax 0x0000000000000030 <+48>: retq By using a rwsem microbenchmark, the down_read_trylock() rate (with a load of 10 to lengthen the lock critical section) on a x86-64 system before and after the patch were: Before Patch After Patch # of Threads rlock rlock ------------ ----- ----- 1 14,496 14,716 2 8,644 8,453 4 6,799 6,983 8 5,664 7,190 On a ARM64 system, the performance results were: Before Patch After Patch # of Threads rlock rlock ------------ ----- ----- 1 23,676 24,488 2 7,697 9,502 4 4,945 3,440 8 2,641 1,603 For the uncontended case (1 thread), the new down_read_trylock() is a little bit faster. For the contended cases, the new down_read_trylock() perform pretty well in x86-64, but performance degrades at high contention level on ARM64. Suggested-by: Linus Torvalds <torvalds@linux-foundation.org> Signed-off-by: Waiman Long <longman@redhat.com> --- kernel/locking/rwsem.h | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) diff --git a/kernel/locking/rwsem.h b/kernel/locking/rwsem.h index 45ee002..1f5775a 100644 --- a/kernel/locking/rwsem.h +++ b/kernel/locking/rwsem.h @@ -174,14 +174,17 @@ static inline int __down_read_killable(struct rw_semaphore *sem) static inline int __down_read_trylock(struct rw_semaphore *sem) { - long tmp; + /* + * Optimize for the case when the rwsem is not locked at all. + */ + long tmp = RWSEM_UNLOCKED_VALUE; - while ((tmp = atomic_long_read(&sem->count)) >= 0) { - if (tmp == atomic_long_cmpxchg_acquire(&sem->count, tmp, - tmp + RWSEM_ACTIVE_READ_BIAS)) { + do { + if (atomic_long_try_cmpxchg_acquire(&sem->count, &tmp, + tmp + RWSEM_ACTIVE_READ_BIAS)) { return 1; } - } + } while (tmp >= 0); return 0; } -- 1.8.3.1 ^ permalink raw reply related [flat|nested] 15+ messages in thread
* Re: [PATCH v4 3/3] locking/rwsem: Optimize down_read_trylock() 2019-02-13 22:00 ` [PATCH v4 3/3] locking/rwsem: Optimize down_read_trylock() Waiman Long @ 2019-02-21 14:14 ` Will Deacon 2019-02-22 3:28 ` Waiman Long 0 siblings, 1 reply; 15+ messages in thread From: Will Deacon @ 2019-02-21 14:14 UTC (permalink / raw) To: Waiman Long Cc: Peter Zijlstra, Ingo Molnar, Thomas Gleixner, linux-kernel, linux-alpha, linux-arm-kernel, linux-c6x-dev, uclinux-h8-devel, linux-hexagon, linux-ia64, linux-m68k, linux-mips, nios2-dev, openrisc, linux-parisc, linuxppc-dev, linux-riscv, linux-s390, linux-sh, sparclinux, linux-um, linux-xtensa, linux-arch, x86, Arnd Bergmann, Borislav On Wed, Feb 13, 2019 at 05:00:17PM -0500, Waiman Long wrote: > Modify __down_read_trylock() to optimize for an unlocked rwsem and make > it generate slightly better code. > > Before this patch, down_read_trylock: > > 0x0000000000000000 <+0>: callq 0x5 <down_read_trylock+5> > 0x0000000000000005 <+5>: jmp 0x18 <down_read_trylock+24> > 0x0000000000000007 <+7>: lea 0x1(%rdx),%rcx > 0x000000000000000b <+11>: mov %rdx,%rax > 0x000000000000000e <+14>: lock cmpxchg %rcx,(%rdi) > 0x0000000000000013 <+19>: cmp %rax,%rdx > 0x0000000000000016 <+22>: je 0x23 <down_read_trylock+35> > 0x0000000000000018 <+24>: mov (%rdi),%rdx > 0x000000000000001b <+27>: test %rdx,%rdx > 0x000000000000001e <+30>: jns 0x7 <down_read_trylock+7> > 0x0000000000000020 <+32>: xor %eax,%eax > 0x0000000000000022 <+34>: retq > 0x0000000000000023 <+35>: mov %gs:0x0,%rax > 0x000000000000002c <+44>: or $0x3,%rax > 0x0000000000000030 <+48>: mov %rax,0x20(%rdi) > 0x0000000000000034 <+52>: mov $0x1,%eax > 0x0000000000000039 <+57>: retq > > After patch, down_read_trylock: > > 0x0000000000000000 <+0>: callq 0x5 <down_read_trylock+5> > 0x0000000000000005 <+5>: xor %eax,%eax > 0x0000000000000007 <+7>: lea 0x1(%rax),%rdx > 0x000000000000000b <+11>: lock cmpxchg %rdx,(%rdi) > 0x0000000000000010 <+16>: jne 0x29 <down_read_trylock+41> > 0x0000000000000012 <+18>: mov %gs:0x0,%rax > 0x000000000000001b <+27>: or $0x3,%rax > 0x000000000000001f <+31>: mov %rax,0x20(%rdi) > 0x0000000000000023 <+35>: mov $0x1,%eax > 0x0000000000000028 <+40>: retq > 0x0000000000000029 <+41>: test %rax,%rax > 0x000000000000002c <+44>: jns 0x7 <down_read_trylock+7> > 0x000000000000002e <+46>: xor %eax,%eax > 0x0000000000000030 <+48>: retq > > By using a rwsem microbenchmark, the down_read_trylock() rate (with a > load of 10 to lengthen the lock critical section) on a x86-64 system > before and after the patch were: > > Before Patch After Patch > # of Threads rlock rlock > ------------ ----- ----- > 1 14,496 14,716 > 2 8,644 8,453 > 4 6,799 6,983 > 8 5,664 7,190 > > On a ARM64 system, the performance results were: > > Before Patch After Patch > # of Threads rlock rlock > ------------ ----- ----- > 1 23,676 24,488 > 2 7,697 9,502 > 4 4,945 3,440 > 8 2,641 1,603 > > For the uncontended case (1 thread), the new down_read_trylock() is a > little bit faster. For the contended cases, the new down_read_trylock() > perform pretty well in x86-64, but performance degrades at high > contention level on ARM64. > > Suggested-by: Linus Torvalds <torvalds@linux-foundation.org> > Signed-off-by: Waiman Long <longman@redhat.com> > --- > kernel/locking/rwsem.h | 13 ++++++++----- > 1 file changed, 8 insertions(+), 5 deletions(-) > > diff --git a/kernel/locking/rwsem.h b/kernel/locking/rwsem.h > index 45ee002..1f5775a 100644 > --- a/kernel/locking/rwsem.h > +++ b/kernel/locking/rwsem.h > @@ -174,14 +174,17 @@ static inline int __down_read_killable(struct rw_semaphore *sem) > > static inline int __down_read_trylock(struct rw_semaphore *sem) > { > - long tmp; > + /* > + * Optimize for the case when the rwsem is not locked at all. > + */ > + long tmp = RWSEM_UNLOCKED_VALUE; > > - while ((tmp = atomic_long_read(&sem->count)) >= 0) { > - if (tmp == atomic_long_cmpxchg_acquire(&sem->count, tmp, > - tmp + RWSEM_ACTIVE_READ_BIAS)) { > + do { > + if (atomic_long_try_cmpxchg_acquire(&sem->count, &tmp, > + tmp + RWSEM_ACTIVE_READ_BIAS)) { > return 1; > } > - } > + } while (tmp >= 0); Nit: but I guess that should be RWSEM_UNLOCKED_VALUE instead of 0. Will ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH v4 3/3] locking/rwsem: Optimize down_read_trylock() 2019-02-21 14:14 ` Will Deacon @ 2019-02-22 3:28 ` Waiman Long 0 siblings, 0 replies; 15+ messages in thread From: Waiman Long @ 2019-02-22 3:28 UTC (permalink / raw) To: Will Deacon Cc: Peter Zijlstra, Ingo Molnar, Thomas Gleixner, linux-kernel, linux-alpha, linux-arm-kernel, linux-c6x-dev, uclinux-h8-devel, linux-hexagon, linux-ia64, linux-m68k, linux-mips, nios2-dev, openrisc, linux-parisc, linuxppc-dev, linux-riscv, linux-s390, linux-sh, sparclinux, linux-um, linux-xtensa, linux-arch, x86, Arnd Bergmann, Borislav On 02/21/2019 09:14 AM, Will Deacon wrote: > On Wed, Feb 13, 2019 at 05:00:17PM -0500, Waiman Long wrote: >> Modify __down_read_trylock() to optimize for an unlocked rwsem and make >> it generate slightly better code. >> >> Before this patch, down_read_trylock: >> >> 0x0000000000000000 <+0>: callq 0x5 <down_read_trylock+5> >> 0x0000000000000005 <+5>: jmp 0x18 <down_read_trylock+24> >> 0x0000000000000007 <+7>: lea 0x1(%rdx),%rcx >> 0x000000000000000b <+11>: mov %rdx,%rax >> 0x000000000000000e <+14>: lock cmpxchg %rcx,(%rdi) >> 0x0000000000000013 <+19>: cmp %rax,%rdx >> 0x0000000000000016 <+22>: je 0x23 <down_read_trylock+35> >> 0x0000000000000018 <+24>: mov (%rdi),%rdx >> 0x000000000000001b <+27>: test %rdx,%rdx >> 0x000000000000001e <+30>: jns 0x7 <down_read_trylock+7> >> 0x0000000000000020 <+32>: xor %eax,%eax >> 0x0000000000000022 <+34>: retq >> 0x0000000000000023 <+35>: mov %gs:0x0,%rax >> 0x000000000000002c <+44>: or $0x3,%rax >> 0x0000000000000030 <+48>: mov %rax,0x20(%rdi) >> 0x0000000000000034 <+52>: mov $0x1,%eax >> 0x0000000000000039 <+57>: retq >> >> After patch, down_read_trylock: >> >> 0x0000000000000000 <+0>: callq 0x5 <down_read_trylock+5> >> 0x0000000000000005 <+5>: xor %eax,%eax >> 0x0000000000000007 <+7>: lea 0x1(%rax),%rdx >> 0x000000000000000b <+11>: lock cmpxchg %rdx,(%rdi) >> 0x0000000000000010 <+16>: jne 0x29 <down_read_trylock+41> >> 0x0000000000000012 <+18>: mov %gs:0x0,%rax >> 0x000000000000001b <+27>: or $0x3,%rax >> 0x000000000000001f <+31>: mov %rax,0x20(%rdi) >> 0x0000000000000023 <+35>: mov $0x1,%eax >> 0x0000000000000028 <+40>: retq >> 0x0000000000000029 <+41>: test %rax,%rax >> 0x000000000000002c <+44>: jns 0x7 <down_read_trylock+7> >> 0x000000000000002e <+46>: xor %eax,%eax >> 0x0000000000000030 <+48>: retq >> >> By using a rwsem microbenchmark, the down_read_trylock() rate (with a >> load of 10 to lengthen the lock critical section) on a x86-64 system >> before and after the patch were: >> >> Before Patch After Patch >> # of Threads rlock rlock >> ------------ ----- ----- >> 1 14,496 14,716 >> 2 8,644 8,453 >> 4 6,799 6,983 >> 8 5,664 7,190 >> >> On a ARM64 system, the performance results were: >> >> Before Patch After Patch >> # of Threads rlock rlock >> ------------ ----- ----- >> 1 23,676 24,488 >> 2 7,697 9,502 >> 4 4,945 3,440 >> 8 2,641 1,603 >> >> For the uncontended case (1 thread), the new down_read_trylock() is a >> little bit faster. For the contended cases, the new down_read_trylock() >> perform pretty well in x86-64, but performance degrades at high >> contention level on ARM64. >> >> Suggested-by: Linus Torvalds <torvalds@linux-foundation.org> >> Signed-off-by: Waiman Long <longman@redhat.com> >> --- >> kernel/locking/rwsem.h | 13 ++++++++----- >> 1 file changed, 8 insertions(+), 5 deletions(-) >> >> diff --git a/kernel/locking/rwsem.h b/kernel/locking/rwsem.h >> index 45ee002..1f5775a 100644 >> --- a/kernel/locking/rwsem.h >> +++ b/kernel/locking/rwsem.h >> @@ -174,14 +174,17 @@ static inline int __down_read_killable(struct rw_semaphore *sem) >> >> static inline int __down_read_trylock(struct rw_semaphore *sem) >> { >> - long tmp; >> + /* >> + * Optimize for the case when the rwsem is not locked at all. >> + */ >> + long tmp = RWSEM_UNLOCKED_VALUE; >> >> - while ((tmp = atomic_long_read(&sem->count)) >= 0) { >> - if (tmp == atomic_long_cmpxchg_acquire(&sem->count, tmp, >> - tmp + RWSEM_ACTIVE_READ_BIAS)) { >> + do { >> + if (atomic_long_try_cmpxchg_acquire(&sem->count, &tmp, >> + tmp + RWSEM_ACTIVE_READ_BIAS)) { >> return 1; >> } >> - } >> + } while (tmp >= 0); > Nit: but I guess that should be RWSEM_UNLOCKED_VALUE instead of 0. > > Will Using RWSEM_UNLOCKED_VALUE may be better. Anyway, it is not a big deal as I am going to change this again in a later patch. Thanks, Longman RWSEM_UNLOCKED_VALUE ^ permalink raw reply [flat|nested] 15+ messages in thread
[parent not found: <20190214103715.O61xiyDUtFfJywghpz2Nwi_R2OMcuxbREoUd59m9Ey4@z>]
* Re: [PATCH v4 0/3] locking/rwsem: Rwsem rearchitecture part 0 [not found] ` <20190214103715.O61xiyDUtFfJywghpz2Nwi_R2OMcuxbREoUd59m9Ey4@z> @ 2019-02-14 10:37 ` Peter Zijlstra [not found] ` <20190214220527.h8SjKk87IPqCmzCDizURd2swYG-vJxuMv01nqE5HhNs@z> [not found] ` <20190215184056.tr5Jo05pLYv3uei7mX7dgqGahK7TOr658G27E5xdbO8@z> 0 siblings, 2 replies; 15+ messages in thread From: Peter Zijlstra @ 2019-02-14 10:37 UTC (permalink / raw) To: Waiman Long Cc: Ingo Molnar, Will Deacon, Thomas Gleixner, linux-kernel, linux-alpha, linux-arm-kernel, linux-c6x-dev, uclinux-h8-devel, linux-hexagon, linux-ia64, linux-m68k, linux-mips, nios2-dev, openrisc, linux-parisc, linuxppc-dev, linux-riscv, linux-s390, linux-sh, sparclinux, linux-um, linux-xtensa, linux-arch, x86, Arnd Bergmann, Borislav Petkov, H. Peter Anvin, Davidlohr Bueso, Linus Torvalds, Andrew Morton, Tim Chen On Wed, Feb 13, 2019 at 05:00:14PM -0500, Waiman Long wrote: > v4: > - Remove rwsem-spinlock.c and make all archs use rwsem-xadd.c. > > v3: > - Optimize __down_read_trylock() for the uncontended case as suggested > by Linus. > > v2: > - Add patch 2 to optimize __down_read_trylock() as suggested by PeterZ. > - Update performance test data in patch 1. > > The goal of this patchset is to remove the architecture specific files > for rwsem-xadd to make it easer to add enhancements in the later rwsem > patches. It also removes the legacy rwsem-spinlock.c file and make all > the architectures use one single implementation of rwsem - rwsem-xadd.c. > > Waiman Long (3): > locking/rwsem: Remove arch specific rwsem files > locking/rwsem: Remove rwsem-spinlock.c & use rwsem-xadd.c for all > archs > locking/rwsem: Optimize down_read_trylock() Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> with the caveat that I'm happy to exchange patch 3 back to my earlier suggestion in case Will expesses concerns wrt the ARM64 performance of Linus' suggestion. ^ permalink raw reply [flat|nested] 15+ messages in thread
[parent not found: <20190214220527.h8SjKk87IPqCmzCDizURd2swYG-vJxuMv01nqE5HhNs@z>]
* Re: [PATCH v4 0/3] locking/rwsem: Rwsem rearchitecture part 0 [not found] ` <20190214220527.h8SjKk87IPqCmzCDizURd2swYG-vJxuMv01nqE5HhNs@z> @ 2019-02-14 22:05 ` Waiman Long 0 siblings, 0 replies; 15+ messages in thread From: Waiman Long @ 2019-02-14 22:05 UTC (permalink / raw) To: Peter Zijlstra Cc: Ingo Molnar, Will Deacon, Thomas Gleixner, linux-kernel, linux-alpha, linux-arm-kernel, linux-c6x-dev, uclinux-h8-devel, linux-hexagon, linux-ia64, linux-m68k, linux-mips, nios2-dev, openrisc, linux-parisc, linuxppc-dev, linux-riscv, linux-s390, linux-sh, sparclinux, linux-um, linux-xtensa, linux-arch, x86, Arnd Bergmann, Borislav Petkov, H. Peter Anvin, Davidlohr Bueso, Linus Torvalds, Andrew Morton, Tim Chen On 02/14/2019 05:37 AM, Peter Zijlstra wrote: > On Wed, Feb 13, 2019 at 05:00:14PM -0500, Waiman Long wrote: >> v4: >> - Remove rwsem-spinlock.c and make all archs use rwsem-xadd.c. >> >> v3: >> - Optimize __down_read_trylock() for the uncontended case as suggested >> by Linus. >> >> v2: >> - Add patch 2 to optimize __down_read_trylock() as suggested by PeterZ. >> - Update performance test data in patch 1. >> >> The goal of this patchset is to remove the architecture specific files >> for rwsem-xadd to make it easer to add enhancements in the later rwsem >> patches. It also removes the legacy rwsem-spinlock.c file and make all >> the architectures use one single implementation of rwsem - rwsem-xadd.c. >> >> Waiman Long (3): >> locking/rwsem: Remove arch specific rwsem files >> locking/rwsem: Remove rwsem-spinlock.c & use rwsem-xadd.c for all >> archs >> locking/rwsem: Optimize down_read_trylock() > Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> > > with the caveat that I'm happy to exchange patch 3 back to my earlier > suggestion in case Will expesses concerns wrt the ARM64 performance of > Linus' suggestion. I inserted a few lock event counters into the rwsem trylock code: static inline int __down_read_trylock(struct rw_semaphore *sem) { /* * Optimize for the case when the rwsem is not locked at all. */ long tmp = RWSEM_UNLOCKED_VALUE; lockevent_inc(rwsem_rtrylock); do { if (atomic_long_try_cmpxchg_acquire(&sem->count, &tmp, tmp + RWSEM_ACTIVE_READ_BIAS)) { rwsem_set_reader_owned(sem); return 1; } lockevent_inc(rwsem_rtrylock_retry); } while (tmp >= 0); lockevent_inc(rwsem_rtrylock_fail); return 0; } static inline int __down_write_trylock(struct rw_semaphore *sem) { long tmp; lockevent_inc(rwsem_wtrylock); tmp = atomic_long_cmpxchg_acquire(&sem->count, RWSEM_UNLOCKED_VALUE, RWSEM_ACTIVE_WRITE_BIAS); if (tmp == RWSEM_UNLOCKED_VALUE) { rwsem_set_owner(sem); return true; } lockevent_inc(rwsem_wtrylock_fail); return false; } I booted the new kernel on a 4-socket 56-core 112-thread Broadwell system. The counter values 1) After bootup: rwsem_rtrylock=784029 rwsem_rtrylock_fail=59 rwsem_rtrylock_retry=394 rwsem_wtrylock=18284 rwsem_wtrylock_fail=230 2) After parallel kernel build (-j112): rwsem_rtrylock=338667559 rwsem_rtrylock_fail=18 rwsem_rtrylock_retry=51 rwsem_wtrylock=17016332 rwsem_wtrylock_fail=98058 At least for these two use cases, try-for-ownership as suggested by Linus is the right choice. Cheers, Longman ^ permalink raw reply [flat|nested] 15+ messages in thread
[parent not found: <20190215184056.tr5Jo05pLYv3uei7mX7dgqGahK7TOr658G27E5xdbO8@z>]
* Re: [PATCH v4 0/3] locking/rwsem: Rwsem rearchitecture part 0 [not found] ` <20190215184056.tr5Jo05pLYv3uei7mX7dgqGahK7TOr658G27E5xdbO8@z> @ 2019-02-15 18:40 ` Will Deacon [not found] ` <20190215185834.QggPIkizUhXeLJZ5LwKZ3hKEMdsrFZ-1d7B16zCNoWs@z> 0 siblings, 1 reply; 15+ messages in thread From: Will Deacon @ 2019-02-15 18:40 UTC (permalink / raw) To: Peter Zijlstra Cc: Waiman Long, Ingo Molnar, Thomas Gleixner, linux-kernel, linux-alpha, linux-arm-kernel, linux-c6x-dev, uclinux-h8-devel, linux-hexagon, linux-ia64, linux-m68k, linux-mips, nios2-dev, openrisc, linux-parisc, linuxppc-dev, linux-riscv, linux-s390, linux-sh, sparclinux, linux-um, linux-xtensa, linux-arch, x86, Arnd Bergmann, Borislav Petkov, H. Peter Anvin, Davidlohr Bueso, Linus Torvalds, Andrew Morton, Tim Chen On Thu, Feb 14, 2019 at 11:37:15AM +0100, Peter Zijlstra wrote: > On Wed, Feb 13, 2019 at 05:00:14PM -0500, Waiman Long wrote: > > v4: > > - Remove rwsem-spinlock.c and make all archs use rwsem-xadd.c. > > > > v3: > > - Optimize __down_read_trylock() for the uncontended case as suggested > > by Linus. > > > > v2: > > - Add patch 2 to optimize __down_read_trylock() as suggested by PeterZ. > > - Update performance test data in patch 1. > > > > The goal of this patchset is to remove the architecture specific files > > for rwsem-xadd to make it easer to add enhancements in the later rwsem > > patches. It also removes the legacy rwsem-spinlock.c file and make all > > the architectures use one single implementation of rwsem - rwsem-xadd.c. > > > > Waiman Long (3): > > locking/rwsem: Remove arch specific rwsem files > > locking/rwsem: Remove rwsem-spinlock.c & use rwsem-xadd.c for all > > archs > > locking/rwsem: Optimize down_read_trylock() > > Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> > > with the caveat that I'm happy to exchange patch 3 back to my earlier > suggestion in case Will expesses concerns wrt the ARM64 performance of > Linus' suggestion. Right, the current proposal doesn't work well for us, unfortunately. Which was your earlier suggestion? Will ^ permalink raw reply [flat|nested] 15+ messages in thread
[parent not found: <20190215185834.QggPIkizUhXeLJZ5LwKZ3hKEMdsrFZ-1d7B16zCNoWs@z>]
* Re: [PATCH v4 0/3] locking/rwsem: Rwsem rearchitecture part 0 [not found] ` <20190215185834.QggPIkizUhXeLJZ5LwKZ3hKEMdsrFZ-1d7B16zCNoWs@z> @ 2019-02-15 18:58 ` Waiman Long 2019-02-18 14:58 ` Will Deacon 0 siblings, 1 reply; 15+ messages in thread From: Waiman Long @ 2019-02-15 18:58 UTC (permalink / raw) To: Will Deacon, Peter Zijlstra Cc: Ingo Molnar, Thomas Gleixner, linux-kernel, linux-alpha, linux-arm-kernel, linux-c6x-dev, uclinux-h8-devel, linux-hexagon, linux-ia64, linux-m68k, linux-mips, nios2-dev, openrisc, linux-parisc, linuxppc-dev, linux-riscv, linux-s390, linux-sh, sparclinux, linux-um, linux-xtensa, linux-arch, x86, Arnd Bergmann, Borislav Petkov, H. Peter Anvin, Davidlohr Bueso, Linus Torvalds, Andrew Morton, Tim Chen On 02/15/2019 01:40 PM, Will Deacon wrote: > On Thu, Feb 14, 2019 at 11:37:15AM +0100, Peter Zijlstra wrote: >> On Wed, Feb 13, 2019 at 05:00:14PM -0500, Waiman Long wrote: >>> v4: >>> - Remove rwsem-spinlock.c and make all archs use rwsem-xadd.c. >>> >>> v3: >>> - Optimize __down_read_trylock() for the uncontended case as suggested >>> by Linus. >>> >>> v2: >>> - Add patch 2 to optimize __down_read_trylock() as suggested by PeterZ. >>> - Update performance test data in patch 1. >>> >>> The goal of this patchset is to remove the architecture specific files >>> for rwsem-xadd to make it easer to add enhancements in the later rwsem >>> patches. It also removes the legacy rwsem-spinlock.c file and make all >>> the architectures use one single implementation of rwsem - rwsem-xadd.c. >>> >>> Waiman Long (3): >>> locking/rwsem: Remove arch specific rwsem files >>> locking/rwsem: Remove rwsem-spinlock.c & use rwsem-xadd.c for all >>> archs >>> locking/rwsem: Optimize down_read_trylock() >> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> >> >> with the caveat that I'm happy to exchange patch 3 back to my earlier >> suggestion in case Will expesses concerns wrt the ARM64 performance of >> Linus' suggestion. > Right, the current proposal doesn't work well for us, unfortunately. Which > was your earlier suggestion? > > Will In my posting yesterday, I showed that most of the trylocks done were actually uncontended. Assuming that pattern hold for the most of the workloads, it will not that bad after all. -Longman ^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [PATCH v4 0/3] locking/rwsem: Rwsem rearchitecture part 0 2019-02-15 18:58 ` Waiman Long @ 2019-02-18 14:58 ` Will Deacon 0 siblings, 0 replies; 15+ messages in thread From: Will Deacon @ 2019-02-18 14:58 UTC (permalink / raw) To: Waiman Long Cc: Peter Zijlstra, Ingo Molnar, Thomas Gleixner, linux-kernel, linux-alpha, linux-arm-kernel, linux-c6x-dev, uclinux-h8-devel, linux-hexagon, linux-ia64, linux-m68k, linux-mips, nios2-dev, openrisc, linux-parisc, linuxppc-dev, linux-riscv, linux-s390, linux-sh, sparclinux, linux-um, linux-xtensa, linux-arch, x86, Arnd Bergmann, Borislav On Fri, Feb 15, 2019 at 01:58:34PM -0500, Waiman Long wrote: > On 02/15/2019 01:40 PM, Will Deacon wrote: > > On Thu, Feb 14, 2019 at 11:37:15AM +0100, Peter Zijlstra wrote: > >> On Wed, Feb 13, 2019 at 05:00:14PM -0500, Waiman Long wrote: > >>> v4: > >>> - Remove rwsem-spinlock.c and make all archs use rwsem-xadd.c. > >>> > >>> v3: > >>> - Optimize __down_read_trylock() for the uncontended case as suggested > >>> by Linus. > >>> > >>> v2: > >>> - Add patch 2 to optimize __down_read_trylock() as suggested by PeterZ. > >>> - Update performance test data in patch 1. > >>> > >>> The goal of this patchset is to remove the architecture specific files > >>> for rwsem-xadd to make it easer to add enhancements in the later rwsem > >>> patches. It also removes the legacy rwsem-spinlock.c file and make all > >>> the architectures use one single implementation of rwsem - rwsem-xadd.c. > >>> > >>> Waiman Long (3): > >>> locking/rwsem: Remove arch specific rwsem files > >>> locking/rwsem: Remove rwsem-spinlock.c & use rwsem-xadd.c for all > >>> archs > >>> locking/rwsem: Optimize down_read_trylock() > >> Acked-by: Peter Zijlstra (Intel) <peterz@infradead.org> > >> > >> with the caveat that I'm happy to exchange patch 3 back to my earlier > >> suggestion in case Will expesses concerns wrt the ARM64 performance of > >> Linus' suggestion. > > Right, the current proposal doesn't work well for us, unfortunately. Which > > was your earlier suggestion? > > > > Will > > In my posting yesterday, I showed that most of the trylocks done were > actually uncontended. Assuming that pattern hold for the most of the > workloads, it will not that bad after all. That's fair enough; if you're going to sit in a tight trylock() loop like the benchmark does, then you're much better off just calling lock() if you care at all about scalability. Will ^ permalink raw reply [flat|nested] 15+ messages in thread
end of thread, other threads:[~2019-02-22 3:28 UTC | newest]
Thread overview: 15+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <20190213220014.kxk4dGPy_-ynCjBbjBru6na4k4ZyVXKT0j8GelBVXk8@z>
2019-02-13 22:00 ` [PATCH v4 0/3] locking/rwsem: Rwsem rearchitecture part 0 Waiman Long
[not found] ` <20190213220015.1ysxH-7CNrdIFhN43HhpwNeF7RSBFoJQPWW5cLRyzjU@z>
2019-02-13 22:00 ` [PATCH v4 1/3] locking/rwsem: Remove arch specific rwsem files Waiman Long
2019-02-20 14:47 ` Will Deacon
[not found] ` <20190213220016.qVZR7YN6b7PNtJaS2f6lV0L261li-233A9oZ44nZLNQ@z>
2019-02-13 22:00 ` [PATCH v4 2/3] locking/rwsem: Remove rwsem-spinlock.c & use rwsem-xadd.c for all archs Waiman Long
[not found] ` <20190214105447.zASTVFPe_Ua4KXhT97AbvJ17RcoLI1qz6hK2MSA1RMk@z>
2019-02-14 10:54 ` Geert Uytterhoeven
[not found] ` <20190214111211.Zz4JLUBLm4iMV2I1EaQtWszvprA6uZ_UN5mNKMeBl04@z>
2019-02-14 11:12 ` Peter Zijlstra
2019-02-20 14:47 ` Will Deacon
[not found] ` <20190213220017.fPHty3eMIFui2_R0HHBvLImq4wVl1Ii36BMjgrdkIxg@z>
2019-02-13 22:00 ` [PATCH v4 3/3] locking/rwsem: Optimize down_read_trylock() Waiman Long
2019-02-21 14:14 ` Will Deacon
2019-02-22 3:28 ` Waiman Long
[not found] ` <20190214103715.O61xiyDUtFfJywghpz2Nwi_R2OMcuxbREoUd59m9Ey4@z>
2019-02-14 10:37 ` [PATCH v4 0/3] locking/rwsem: Rwsem rearchitecture part 0 Peter Zijlstra
[not found] ` <20190214220527.h8SjKk87IPqCmzCDizURd2swYG-vJxuMv01nqE5HhNs@z>
2019-02-14 22:05 ` Waiman Long
[not found] ` <20190215184056.tr5Jo05pLYv3uei7mX7dgqGahK7TOr658G27E5xdbO8@z>
2019-02-15 18:40 ` Will Deacon
[not found] ` <20190215185834.QggPIkizUhXeLJZ5LwKZ3hKEMdsrFZ-1d7B16zCNoWs@z>
2019-02-15 18:58 ` Waiman Long
2019-02-18 14:58 ` Will Deacon
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox