Greetings,

I got the below dmesg and the first bad commit is

commit a21813be23329e2788164eab532e79cb0e513cfc
Author: Nathan Zimmer <nzimmer@sgi.com>
Date:   Sat Mar 2 15:25:37 2013 +1100

    procfs: improve scaling in proc
    
    I am currently tracking a hotlock reported by a customer on a large
    system, 512 cores.  I am currently running 3.8-rc7 but the issue looks
    like it has been this way for a very long time.  The offending lock is
    proc_dir_entry->pde_unload_lock.
    
    This patch converts the lock to use rcu.  However the pde_openers list
    still is controlled by a spin lock.  I tested on a 4096 machine and the
    lock doesn't seem hot at least according to perf.
    
    This is a refresh of what was orignally suggested by Eric Dumazet some
    time ago.  I have also taken in some comments from Andrew and several
    other people whose names escape me but I am quite grateful too.
    
    Supporting numbers, lower is better, they are from the test I posted earlier.
    cpuinfo baseline        Rcu
    tasks   read-sec        read-sec
    1       0.0141          0.0141
    2       0.0140          0.0142
    4       0.0140          0.0141
    8       0.0145          0.0140
    16      0.0553          0.0168
    32      0.1688          0.0549
    64      0.5017          0.1690
    128     1.7005          0.5038
    256     5.2513          2.0804
    512     8.0529          3.0162
    
    Signed-off-by: Nathan Zimmer <nzimmer@sgi.com>
    Cc: "Eric W. Biederman" <ebiederm@xmission.com>
    Cc: Eric Dumazet <eric.dumazet@gmail.com>
    Cc: Alexander Viro <viro@zeniv.linux.org.uk>
    Cc: David Woodhouse <dwmw2@infradead.org>
    Cc: Alexey Dobriyan <adobriyan@gmail.com>
    Cc: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
    Signed-off-by: Andrew Morton <akpm@linux-foundation.org>

[   78.994163] atkbd: probe of serio1 rejects match -19
[   79.196148] Freeing unused kernel memory: 664k freed
[   80.786460] BUG: sleeping function called from invalid context at /c/kernel-tests/src/tip/mm/slub.c:925
[   80.787886] in_atomic(): 0, irqs_disabled(): 0, pid: 29, name: modprobe
[   80.789636] 2 locks held by modprobe/29:
[   80.790343]  #0:  (rcu_read_lock){.+.+.+}, at: [<ffffffff8117347b>] rcu_read_lock+0x0/0xbc
[   80.794346]  #1:  (rcu_read_lock){.+.+.+}, at: [<ffffffff8117347b>] rcu_read_lock+0x0/0xbc
[   80.797089] Pid: 29, comm: modprobe Not tainted 3.9.0-rc1-01618-gdf2363d #223
[   80.798113] Call Trace:
[   80.798856]  [<ffffffff8106fc60>] __might_sleep+0x266/0x275
[   80.799830]  [<ffffffff81146311>] ? single_open+0x30/0xb6
[   80.801733]  [<ffffffff811132fd>] slab_pre_alloc_hook.isra.48+0x42/0x4f
[   80.802779]  [<ffffffff81114f7b>] kmem_cache_alloc+0x2a/0xf6
[   80.803746]  [<ffffffff8117e899>] ? cmdline_proc_open+0x25/0x25
[   80.805383]  [<ffffffff81146311>] single_open+0x30/0xb6
[   80.806281]  [<ffffffff810c05df>] ? rcu_is_cpu_idle+0x42/0x8a
[   80.807242]  [<ffffffff8114556d>] ? seq_release_private+0x59/0x59
[   80.808865]  [<ffffffff8117e890>] cmdline_proc_open+0x1c/0x25
[   80.809811]  [<ffffffff811738e6>] proc_reg_open+0x128/0x1e3
[   80.810684]  [<ffffffff811737be>] ? proc_reg_release+0x246/0x246
[   80.811602]  [<ffffffff81119d82>] do_dentry_open+0x228/0x34a
[   80.813260]  [<ffffffff8112a226>] ? __inode_permission+0x115/0x12b
[   80.814223]  [<ffffffff81119f00>] finish_open+0x5c/0x7a
[   80.815056]  [<ffffffff8112b5bb>] do_last.isra.16+0xbcc/0xecb
[   80.815945]  [<ffffffff8112a2bd>] ? inode_permission+0x81/0x8a
[   80.817660]  [<ffffffff8112cfa6>] path_openat.isra.17+0xf0/0x494
[   80.818623]  [<ffffffff8112e7c6>] do_filp_open+0x3a/0xad
[   80.819475]  [<ffffffff8113eef6>] ? __alloc_fd+0x25e/0x277
[   80.821142]  [<ffffffff8111b4dd>] do_sys_open+0x91/0x17b
[   80.822065]  [<ffffffff8111b5f2>] SyS_open+0x2b/0x3b
[   80.822873]  [<ffffffff813d6f6b>] system_call_fastpath+0x1a/0x1f
[   80.827403] 
[   80.827765] ================================================

git bisect start df2363d9ad7ffd43b5e86604c7a28f4adc6bd75e 725a7c316ea0d1329b630e31f82d0a2e762e0a65 --
git bisect good 62159550cf357e60632a7ac147b4c66b1e2b0630  #    18  2013-03-07 21:37:08  freezer: do not send a fake signal to a PF_DUMPCORE thread
git bisect  bad bcbe478b5a2c1c5831dbe1c31d4b6a7d7438ccee  #     0  2013-03-07 21:39:14  aio: kill struct aio_ring_info
git bisect  bad 93102c910da3c99169b97ed03fdd53c64eadc284  #     0  2013-03-07 21:41:16  gadget: remove only user of aio retry
git bisect  bad df2c264613410a2f59d4f13082207d5e4c0381fe  #     0  2013-03-07 21:43:13  ipc: set EFAULT as default error in load_msg()
git bisect  bad 1b716364563aa3fcd04eddea6fe08bd49574ec7f  #     0  2013-03-07 21:44:39  kexec: Use min() and min_t() to simplify logic
git bisect  bad a21813be23329e2788164eab532e79cb0e513cfc  #     0  2013-03-07 21:46:14  procfs: improve scaling in proc
git bisect good c391a36c9b415f3276d1059594a635e66d3f5d7a  #    12  2013-03-07 21:47:58  coredump: make wait_for_dump_helpers() freezable
git bisect good c391a36c9b415f3276d1059594a635e66d3f5d7a  #    38  2013-03-07 21:49:41  coredump: make wait_for_dump_helpers() freezable
git bisect  bad df2363d9ad7ffd43b5e86604c7a28f4adc6bd75e  #     0  2013-03-07 21:51:39  kernel/sys.c: make prctl(PR_SET_MM) generally available
git bisect good 91ce46c0d79238fc84ee5f8c0936d4dcb66d1fd3  #    31  2013-03-07 21:54:23  Revert "procfs: improve scaling in proc"
git bisect  bad 9edbffb58ae00067e264ef70d5141c1d85049029  #     0  2013-03-07 21:57:05  Add linux-next specific files for 20130307

Thanks,
Fengguang