Greetings, I got the below dmesg and the first bad commit is commit 1f6587114a689a5d7fdfb0d4abc818117e3182a5 Author: Davidlohr Bueso Date: Thu Jun 6 10:41:56 2013 +1000 ipc: move rcu lock out of ipc_addid This patchset continues the work that began in the sysv ipc semaphore scaling series: https://lkml.org/lkml/2013/3/20/546 Just like semaphores used to be, sysv shared memory and msg queues also abuse the ipc lock, unnecessarily holding it for operations such as permission and security checks. This patchset mostly deals with mqueues, and while shared mem can be done in a very similar way, I want to get these patches out in the open first. It also does some pending cleanups, mostly focused on the two level locking we have in ipc code, taking care of ipc_addid() and ipcctl_pre_down_nolock() - yes there are still functions that need to be updated as well. This patch: Make all callers explicitly take and release the RCU read lock. This addresses the two level locking seen in newary(), newseg() and newqueue(). For the last two, explicitly unlock the ipc object and the rcu lock, instead of calling the custom shm_unlock and msg_unlock functions. The next patch will deal with the open coded locking for ->perm.lock Signed-off-by: Davidlohr Bueso Cc: Andi Kleen Cc: Rik van Riel Signed-off-by: Andrew Morton [ 51.524946] [ 51.525983] =============================== [ 51.532875] [ INFO: suspicious RCU usage. ] [ 51.535385] 3.10.0-rc4-next-20130606 #6 Not tainted [ 51.538304] ------------------------------- [ 51.540937] /c/kernel-tests/src/stable/include/linux/rcupdate.h:471 Illegal context switch in RCU read-side critical section! [ 51.548110] [ 51.548110] other info that might help us debug this: [ 51.548110] [ 51.553055] [ 51.553055] rcu_scheduler_active = 1, debug_locks = 1 [ 51.557199] 2 locks held by trinity/1107: [ 51.560168] #0: (&ids->rw_mutex){+.+.+.}, at: [] ipcget+0x38/0x2b3 [ 51.566465] #1: (rcu_read_lock){.+.+..}, at: [] newseg+0x19d/0x3fd [ 51.572413] [ 51.572413] stack backtrace: [ 51.574761] CPU: 0 PID: 1107 Comm: trinity Not tainted 3.10.0-rc4-next-20130606 #6 [ 51.579331] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2007 [ 51.583068] 0000000000000001 ffff880004a07d88 ffffffff817b1f5c ffff880004a07db8 [ 51.592119] ffffffff810f2f1d ffffffff81b78569 00000000000001a8 0000000000000000 [ 51.596726] 0000000000000000 ffff880004a07de8 ffffffff810ded5e ffff880004a07fd8 [ 51.605189] Call Trace: [ 51.606409] [] dump_stack+0x19/0x1b [ 51.609632] [] lockdep_rcu_suspicious+0xeb/0xf4 [ 51.612905] [] __might_sleep+0x59/0x1dc [ 51.618614] [] idr_preload+0x9b/0x142 [ 51.621939] [] ipc_addid+0x3d/0x193 [ 51.624373] [] newseg+0x221/0x3fd [ 51.626596] [] ? newseg+0x19d/0x3fd [ 51.630177] [] ipcget+0x1be/0x2b3 [ 51.633174] [] ? retint_swapgs+0x13/0x1b [ 51.636356] [] SyS_shmget+0x59/0x5d [ 51.639576] [] ? shm_try_destroy_orphaned+0xbf/0xbf [ 51.643673] [] ? shm_get_unmapped_area+0x20/0x20 [ 51.647321] [] ? shm_security+0xb/0xb [ 51.650831] [] system_call_fastpath+0x16/0x1b git bisect start 4e1e7059d375482daeeda395bba2939679b1ee14 e3e160d1c8b68beede8e47b281bf6369b833f1c5 -- git bisect good 04eb3039eba693d510952e8867fc0b057955f840 # 10 2013-06-06 19:18:45 shrinker: convert remaining shrinkers to count/scan API git bisect good 870ce17d96260246bd343f55b55b6adb468aab76 # 10 2013-06-06 19:46:15 rtc: rtc-mpc5121: use devm_*() functions git bisect good e75aff7f353f503d38863fd0a3e1d1e4a3312325 # 10 2013-06-06 19:52:37 x86: kill TIF_DEBUG git bisect bad 93f94fca3e3edc76bd3f392edcd8ddc13629c626 # 0 2013-06-06 19:58:13 ipc/sem.c: rename try_atomic_semop() to perform_atomic_semop(), docu update git bisect good e6391c3a76e29b92a6c0183f4d94776c1ca9ecbc # 10 2013-06-06 20:45:47 ia64: remove setting for saved_max_pfn git bisect bad 47cde6c8674ec0598b6da4c0813ec984ee209d97 # 0 2013-06-06 20:50:16 ipc,msg: shorten critical region in msgctl_down git bisect good 343ff671539e6ce9bc849d881fcec5c004342e0a # 10 2013-06-06 20:55:29 ipc/shmc.c: eliminate ugly 80-col tricks git bisect bad e3a7780fbb54d93fd053182d4b1f8c5596b37247 # 0 2013-06-06 21:02:39 ipc: introduce ipc object locking helpers git bisect bad 1f6587114a689a5d7fdfb0d4abc818117e3182a5 # 0 2013-06-06 21:12:01 ipc: move rcu lock out of ipc_addid git bisect good 343ff671539e6ce9bc849d881fcec5c004342e0a # 30 2013-06-06 21:15:37 ipc/shmc.c: eliminate ugly 80-col tricks git bisect bad 4e1e7059d375482daeeda395bba2939679b1ee14 # 0 2013-06-06 21:15:40 Add linux-next specific files for 20130606 git bisect bad 4e1e7059d375482daeeda395bba2939679b1ee14 # 0 2013-06-06 21:15:52 Add linux-next specific files for 20130606 Thanks, Fengguang