Greetings, I got the below dmesg and the first bad commit is commit f284b77333b1320491f138b76f11006bbc962569 Author: Tejun Heo Date: Wed Jun 12 13:50:58 2013 -0700 cgroup: use percpu refcnt for cgroup_subsys_states A css (cgroup_subsys_state) is how each cgroup is represented to a controller. As such, it can be used in hot paths across the various subsystems different controllers are associated with. One of the common operations is reference counting, which up until now has been implemented using a global atomic counter and can have significant adverse impact on scalability. For example, css refcnt can be gotten and put multiple times by blkcg for each IO request. For highops configurations which try to do as much per-cpu as possible, the global frequent refcnting can be very expensive. In general, given the various hugely diverse paths css's end up being used from, we need to make it cheap and highly scalable. In its usage, css refcnting isn't very different from module refcnting. This patch converts css refcnting to use the recently added percpu_ref. css_get/tryget/put() directly maps to the matching percpu_ref operations and the deactivation logic is no longer necessary as percpu_ref already has refcnt killing. The only complication is that as the refcnt is per-cpu, percpu_ref_kill() in itself doesn't ensure that further tryget operations will fail, which we need to guarantee before invoking ->css_offline()'s. This is resolved collecting kill confirmation using percpu_ref_kill_and_confirm() and initiating the offline phase of destruction after all css refcnt's are confirmed to be seen as killed on all CPUs. The previous patches already splitted destruction into two phases, so percpu_ref_kill_and_confirm() can be hooked up easily. This patch removes css_refcnt() which is used for rcu dereference sanity check in css_id(). While we can add a percpu refcnt API to ask the same question, css_id() itself is scheduled to be removed fairly soon, so let's not bother with it. Just drop the sanity check and use rcu_dereference_raw() instead. Signed-off-by: Tejun Heo Cc: Michal Hocko Cc: Mike Snitzer Cc: Vivek Goyal Cc: "Alasdair G. Kergon" Cc: Jens Axboe Cc: Mikulas Patocka Cc: Glauber Costa Cc: Kent Overstreet [ 0.000000] Initializing cgroup subsys cpuset [ 0.000000] ------------[ cut here ]------------ [ 0.000000] WARNING: at /c/kernel-tests/src/tip/mm/percpu.c:292 pcpu_mem_zalloc+0x2f/0x80() [ 0.000000] CPU: 0 PID: 0 Comm: swapper Not tainted 3.10.0-rc4-00064-gf284b773 #57 [ 0.000000] 0000000000000009 ffffffff82133d58 ffffffff81c259e5 ffffffff82133d98 [ 0.000000] ffffffff810818c6 ffffffff82133da8 0000000000000000 0000000000000004 [ 0.000000] 0000000000000082 0000000000000004 0000000000000004 ffffffff82133da8 [ 0.000000] Call Trace: [ 0.000000] [] dump_stack+0x19/0x1b [ 0.000000] [] warn_slowpath_common+0x66/0x90 [ 0.000000] [] warn_slowpath_null+0x15/0x20 [ 0.000000] [] pcpu_mem_zalloc+0x2f/0x80 [ 0.000000] [] pcpu_alloc+0x28d/0xb30 [ 0.000000] [] ? lock_acquired+0x2ae/0x2f0 [ 0.000000] [] ? trace_hardirqs_off_caller+0x38/0xc0 [ 0.000000] [] ? cgroup_pidlist_release+0x50/0x50 [ 0.000000] [] __alloc_percpu+0xb/0x10 [ 0.000000] [] percpu_ref_init+0x2b/0x60 [ 0.000000] [] init_cgroup_css.isra.24+0x30/0xc0 [ 0.000000] [] cgroup_init_subsys+0x79/0xe9 [ 0.000000] [] cgroup_init_early+0x1af/0x1bf [ 0.000000] [] start_kernel+0x1e/0x4a8 [ 0.000000] [] ? memblock_reserve+0x46/0x4b [ 0.000000] [] x86_64_start_reservations+0x2a/0x2c [ 0.000000] [] x86_64_start_kernel+0xec/0xfb [ 0.000000] ---[ end trace 4eaa2a86a8e2da22 ]--- git bisect start f284b77333b1320491f138b76f11006bbc962569 f722406faae2d073cc1d01063d1123c35425939e -- git bisect good c7153d0643dd161df9e7a2fcb952418055aa5fca # 09:39 60+ Merge tag 'driver-core-3.10-rc2' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/driver-core git bisect good 088d812fe97a4cc8edaed69f91511ea063b14cef # 09:54 60+ Merge tag 'for-linus-v3.10-rc3' of git://oss.sgi.com/xfs/xfs git bisect good 977b55cf988b86cdc929fef730a78aca98a8ebdb # 10:19 60+ Merge tag 'please-pull-aertracefix' of git://git.kernel.org/pub/scm/linux/kernel/git/ras/ras git bisect good aa3ae6de36c86994c3446bdbc6b69ece9193732a # 10:33 60+ Merge branch 'merge' of git://git.kernel.org/pub/scm/linux/kernel/git/benh/powerpc git bisect good 0ab60871b451b857206d7afc0d3033be866f76b4 # 11:01 60+ Merge tag 'jfs-3.10-rc5' of git://github.com/kleikamp/linux-shaggy git bisect good de3c0749e2c1960afcc433fc5da136b85c8bd896 # 11:34 60+ percpu-refcount: implement percpu_tryget() along with percpu_ref_kill_and_confirm() git bisect good cc5943a7816ba6c00639837a62131386619548dc # 12:04 60+ cgroup: mark "notify_on_release" and "release_agent" cgroup files insane git bisect good b5f1b7123762411e6a432f8236304e4dcaf5b4d8 # 12:26 60+ cgroup: use kzalloc() and list_del_init() git bisect good 919e0c4a614fce2fdf8a57e5500841e18bb523e6 # 12:36 60+ cgroup: drop unnecessary RCU dancing from __put_css_set() git bisect good fbcc713b2b109bd180688b65c3967d5a1cd46010 # 12:52 60+ cgroup: reorder the operations in cgroup_destroy_locked() git bisect good 903c63033fff6a0f2f60b7c71831fe3b25ea9abc # 15:34 60+ cgroup: split cgroup destruction into two steps git bisect good 903c63033fff6a0f2f60b7c71831fe3b25ea9abc # 19:12 180+ cgroup: split cgroup destruction into two steps git bisect bad f284b77333b1320491f138b76f11006bbc962569 # 19:12 0- cgroup: use percpu refcnt for cgroup_subsys_states git bisect good a2c764c7dbc632ab6ac48b2d833f44dce26ba1ed # 22:43 180+ Revert "cgroup: use percpu refcnt for cgroup_subsys_states" git bisect good 26e04462c8b78d079d3231396ec72d58a14f114b # 00:32 180+ Merge git://git.kernel.org/pub/scm/linux/kernel/git/davem/net git bisect good c04efed734409f5a44715b54a6ca1b54b0ccf215 # 03:51 180+ Add linux-next specific files for 20130607 Thanks, Fengguang