Greetings, I got the below dmesg and the first bad commit is git://git.linaro.org/people/dlezcano/linux sched/idle-balance commit e7b0e894d633a59a5de296b1ec45806993239799 Author: Peter Zijlstra AuthorDate: Sat Feb 11 06:05:00 2012 +0100 Commit: Daniel Lezcano CommitDate: Wed Feb 5 11:55:42 2014 +0100 sched/fair: track cgroup depth Track depth in cgroup tree, this is useful for things like find_matching_se() where you need to get to a common parent of two sched entities. Keeping the depth avoids having to calculate it on the spot, which saves a number of possible cache-misses. Signed-off-by: Peter Zijlstra Link: http://lkml.kernel.org/r/1328936700.2476.17.camel@laptop Signed-off-by: Daniel Lezcano +---------------------------------------------------------+----+ | | | +---------------------------------------------------------+----+ | boot_successes | 0 | | boot_failures | 19 | | BUG:unable_to_handle_kernel_NULL_pointer_dereference_at | 19 | | Oops:SMP | 19 | | RIP:pick_next_task_fair | 7 | | Kernel_panic-not_syncing:Fatal_exception | 19 | | backtrace:do_group_exit | 19 | | backtrace:SyS_exit_group | 19 | | RIP:check_preempt_wakeup | 12 | +---------------------------------------------------------+----+ [ 11.140305] Freeing unused kernel memory: 648K (ffff88000155e000 - ffff880001600000) [ 11.156403] Freeing unused kernel memory: 1852K (ffff880001831000 - ffff880001a00000) [ 11.177538] random: init urandom read with 5 bits of entropy available [ 11.193582] BUG: unable to handle kernel NULL pointer dereference at 0000000000000078 [ 11.194018] IP: [] pick_next_task_fair+0x171/0x238 [ 11.194018] PGD 0 [ 11.194018] Oops: 0000 [#1] SMP [ 11.194018] Modules linked in: [ 11.194018] CPU: 0 PID: 83 Comm: hostname Not tainted 3.14.0-rc1-00092-g92ed244 #33 [ 11.194018] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 [ 11.194018] task: ffff880007bea840 ti: ffff88000ef9e000 task.ti: ffff88000ef9e000 [ 11.194018] RIP: 0010:[] [] pick_next_task_fair+0x171/0x238 [ 11.194018] RSP: 0018:ffff88000ef9fd70 EFLAGS: 00010046 [ 11.194018] RAX: 00000002893d47ce RBX: ffff88000ee6f248 RCX: 0000000000000000 [ 11.194018] RDX: 0000000000000000 RSI: ffff88000ef053d8 RDI: ffff88000d928078 [ 11.194018] RBP: ffff88000ef9fda0 R08: 000000000000297c R09: 000000000000b6c8 [ 11.194018] R10: ffff88000e60a000 R11: 00000000000023d5 R12: ffff88000d928000 [ 11.194018] R13: ffff88000e6127b8 R14: 0000000000000000 R15: 0000000000000001 [ 11.194018] FS: 0000000000000000(0000) GS:ffff88000e600000(0000) knlGS:0000000000000000 [ 11.194018] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 11.194018] CR2: 0000000000000078 CR3: 0000000001a06000 CR4: 00000000000006f0 [ 11.194018] Stack: [ 11.194018] ffff88000e612740 ffff88000e612740 0000000000000000 ffff880007beac38 [ 11.194018] ffff88000d928000 ffff880007bea840 ffff88000ef9fea0 ffffffff8154fa9a [ 11.194018] ffff88000ef9ffd8 ffff880007bea840 0000000000012740 0000000000000011 [ 11.194018] Call Trace: [ 11.194018] [] __schedule+0x1be/0x756 [ 11.194018] [] ? _raw_spin_unlock_irqrestore+0x51/0x7b [ 11.194018] [] ? do_notify_parent+0x22b/0x240 [ 11.194018] [] ? task_function_call+0x44/0x4e [ 11.194018] [] ? __perf_event_task_sched_out+0x329/0x329 [ 11.194018] [] ? _raw_spin_lock_irqsave+0x25/0x69 [ 11.194018] [] schedule+0x6f/0x71 [ 11.194018] [] do_exit+0x8e8/0x927 [ 11.194018] [] ? vtime_account_user+0x52/0x57 [ 11.194018] [] do_group_exit+0x4a/0x110 [ 11.194018] [] SyS_exit_group+0x14/0x14 [ 11.194018] [] tracesys+0xdd/0xe2 [ 11.194018] Code: e8 f2 d2 ff ff 4c 8b a3 80 00 00 00 4d 85 e4 0f 85 e8 fe ff ff 4c 8d 63 98 4d 39 e6 74 61 4d 85 f6 74 5c 49 83 c6 68 4c 8b 6b 78 <49> 8b 7e 78 49 39 fd 74 30 44 8b 7b 68 45 8b 6e 68 45 39 ef 7f [ 11.194018] RIP [] pick_next_task_fair+0x171/0x238 [ 11.194018] RSP [ 11.194018] CR2: 0000000000000078 [ 11.194018] ---[ end trace d8c5bf7ff957e138 ]--- [ 11.194018] Kernel panic - not syncing: Fatal exception git bisect start 92ed2443d0f6e23e9594bb601be3168e272749a0 38dbfb59d1175ef458d006556061adeaa8751b72 -- git bisect good 7b13b156348b40fec269e870b16aa08dbb498e8d # 18:27 20+ 0 Merge 'spi/for-next' into devel-cairo-x86_64-201402052359 git bisect bad 9fa2e2a660ea38b0d0dc750e804377a81faab387 # 19:20 0- 2 Merge 'dlezcano/sched/idle-balance' into devel-cairo-x86_64-201402052359 git bisect bad 7ae0268f44c13f451ffdb8577e60e54319d06f0b # 20:10 0- 10 sched/fair: clean up __clear_buddies_* git bisect good 58b46da336a9312b2e21bb576d1c2c484dbf6257 # 20:52 20+ 0 sched/numa: Rename variables in task_numa_fault() git bisect good a68f0647428f0adba291ec07bbfad1b6ae68f56b # 21:36 20+ 0 sched: Fix race in idle_balance() git bisect good a73b2fc4f74ca24e7d9261d6728a554227b7c913 # 22:05 20+ 0 sched: Clean up idle task SMP logic git bisect bad 2c9b2540b77a3f7d25ef63f2449a9edf9054dd4e # 22:47 0- 1 sched: Push put_prev_task() into pick_next_task() git bisect bad e7b0e894d633a59a5de296b1ec45806993239799 # 23:09 0- 8 sched/fair: track cgroup depth # first bad commit: [e7b0e894d633a59a5de296b1ec45806993239799] sched/fair: track cgroup depth git bisect good a73b2fc4f74ca24e7d9261d6728a554227b7c913 # 23:27 60+ 0 sched: Clean up idle task SMP logic git bisect bad 92ed2443d0f6e23e9594bb601be3168e272749a0 # 23:27 0- 19 0day head guard for 'devel-cairo-x86_64-201402052359' git bisect good ef42c58a5b4b8060a3931aab36bf2b4f81b44afc # 00:59 60+ 0 Merge branch 'irq-core-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip git bisect good 0cc2aa51be9d2f2b001c0e070b2e5cdde89b39f4 # 01:55 60+ 0 Add linux-next specific files for 20140206 Thanks, Fengguang