Greetings, 0day kernel testing robot got the below dmesg and the first bad commit is git://git.kernel.org/pub/scm/linux/kernel/git/paulmck/linux-rcu.git rcu/exp commit 86d8f10e35c1e63d5a839792efd7c3cb6a564fe4 Author: Paul E. McKenney AuthorDate: Fri Aug 22 14:03:38 2014 -0700 Commit: Paul E. McKenney CommitDate: Wed Aug 27 10:00:30 2014 -0700 rcutorture: Add preemption-slam testing This commit adds kernel threads whose sole purpose is to add random preemption of the code under test. It adds four kernel boot parameters to control the number of preemption-slam kthreads, their real-time priority, the length of time that they sleep, and the length of time that they spin at the real-time priority. Signed-off-by: Paul E. McKenney =================================================== PARENT COMMIT NOT CLEAN. LOOK OUT FOR WRONG BISECT! =================================================== Attached dmesg for the parent commit, too, to help confirm whether it is a noise error. +------------------------------------------------+------------+------------+------------------+ | | b52938be99 | 86d8f10e35 | v3.17-rc2_082810 | +------------------------------------------------+------------+------------+------------------+ | boot_successes | 112 | 19 | 0 | | boot_failures | 8 | 9 | 11 | | BUG:kernel_boot_crashed | 6 | | | | BUG:kernel_boot_hang | 2 | | | | RIP:__might_sleep | 0 | 2 | 7 | | Kernel_panic-not_syncing:softlockup:hung_tasks | 0 | 9 | 11 | | backtrace:stutter_wait | 0 | 2 | 5 | | RIP:_cond_resched | 0 | 3 | 1 | | backtrace:trace_apic_timer_interrupt | 0 | 1 | | | RIP:stutter_wait | 0 | 4 | 2 | | backtrace:apic_timer_interrupt | 0 | 6 | 5 | | RIP:function_test_events_call | 0 | 0 | 1 | | backtrace:ftrace_call | 0 | 0 | 1 | +------------------------------------------------+------------+------------+------------------+ [ 9.123983] Testing event mm_vmscan_direct_reclaim_end: [ 32.110003] NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [torture_slam_kt:115] [ 32.110003] CPU: 1 PID: 115 Comm: torture_slam_kt Not tainted 3.17.0-rc2-00063-g86d8f10 #7 [ 32.110003] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 [ 32.110003] task: ffff88000b395650 ti: ffff88000b210000 task.ti: ffff88000b210000 [ 32.110003] RIP: 0010:[] [] __might_sleep+0x18f/0x1a0 [ 32.110003] RSP: 0000:ffff88000b213e10 EFLAGS: 00000202 [ 32.110003] RAX: ffff88000b395650 RBX: 0000000000000000 RCX: 00000000000002a0 [ 32.110003] RDX: 0000000000000073 RSI: 0000000000000276 RDI: ffffffff82da8b58 [ 32.110003] RBP: ffff88000b213e10 R08: 0000000000000ae8 R09: 0000000000000000 [ 32.110003] R10: 0000000000000000 R11: 0000000000000000 R12: ffff88000b213df0 [ 32.110003] R13: ffff88000b213e10 R14: ffffffff8240723f R15: ffff88000b213d80 [ 32.110003] FS: 0000000000000000(0000) GS:ffff880012400000(0000) knlGS:0000000000000000 [ 32.110003] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 32.110003] CR2: 0000000000000000 CR3: 0000000003012000 CR4: 00000000000406e0 [ 32.110003] Stack: [ 32.110003] ffff88000b213e28 ffffffff8120075b ffff880009048c90 ffff88000b213e48 [ 32.110003] ffffffff81201063 0000006300000000 ffff880010e48300 ffff88000b213f48 [ 32.110003] ffffffff81105150 0000000000000000 ffff88000b395650 0000000000000000 [ 32.110003] Call Trace: [ 32.110003] [] stutter_wait+0x9b/0x130 [ 32.110003] [] torture_slam_kthread+0xe3/0x190 [ 32.110003] [] kthread+0x130/0x160 [ 32.110003] [] ? __kthread_unpark+0xa0/0xa0 [ 32.110003] [] ret_from_fork+0x7c/0xb0 [ 32.110003] [] ? __kthread_unpark+0xa0/0xa0 [ 32.110003] Code: 44 00 00 48 83 05 c1 e5 a3 02 01 f6 c4 02 0f 84 ae fe ff ff 65 48 8b 04 25 00 b9 00 00 8b 90 b0 03 00 00 48 83 05 a9 e5 a3 02 01 <85> d2 0f 84 8f fe ff ff 5d c3 0f 1f 80 00 00 00 00 0f 1f 44 00 [ 32.110003] Kernel panic - not syncing: softlockup: hung tasks [ 32.110003] CPU: 1 PID: 115 Comm: torture_slam_kt Tainted: G L 3.17.0-rc2-00063-g86d8f10 #7 [ 32.110003] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 [ 32.110003] 0000000000000007 ffff880012403df8 ffffffff823f8f74 ffffffff82d98e85 [ 32.110003] ffff880012403e70 ffffffff823f0db6 0000000000000008 ffff880012403e80 [ 32.110003] ffff880012403e20 0000000000000001 0000000000000000 ffffffff811504f3 [ 32.110003] Call Trace: [ 32.110003] [] dump_stack+0x85/0xba [ 32.110003] [] panic+0x110/0x300 [ 32.110003] [] ? console_unlock+0x373/0x680 [ 32.110003] [] watchdog_timer_fn+0x283/0x2e0 [ 32.110003] [] hrtimer_run_queues+0x159/0x470 [ 32.110003] [] ? watchdog+0x40/0x40 [ 32.110003] [] update_process_times+0x42/0xc0 [ 32.110003] [] tick_periodic+0x38/0x130 [ 32.110003] [] tick_handle_periodic+0x2e/0xb0 [ 32.110003] [] ? torture_stutter+0x1d0/0x1d0 [ 32.110003] [] local_apic_timer_interrupt+0x3f/0x80 [ 32.110003] [] smp_apic_timer_interrupt+0x5f/0x80 [ 32.110003] [] apic_timer_interrupt+0x6d/0x80 [ 32.110003] [] ? __might_sleep+0x18f/0x1a0 [ 32.110003] [] stutter_wait+0x9b/0x130 [ 32.110003] [] torture_slam_kthread+0xe3/0x190 [ 32.110003] [] kthread+0x130/0x160 [ 32.110003] [] ? __kthread_unpark+0xa0/0xa0 [ 32.110003] [] ret_from_fork+0x7c/0xb0 [ 32.110003] [] ? __kthread_unpark+0xa0/0xa0 [ 32.110003] Kernel Offset: 0x0 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffff9fffffff) Elapsed time: 40 git bisect start 8153a6bbcc340b292248b66a6db89b991fe23995 52addcf9d6669fa439387610bc65c92fa0980cef -- git bisect good fa5b2a0d6039cf49777a5dc734820b5b4935d569 # 10:55 25+ 0 Merge 'spi/fix/dw' into devel-hourly-2014082810 git bisect bad bab6cbcfc9ff6b0abb1506a6c42e93765f3c5162 # 11:04 11- 20 Merge 'arm-soc/for-next' into devel-hourly-2014082810 git bisect good 61ca93aaede4e63c82b1ce60cddab0f922290115 # 11:09 28+ 4 Merge 'kvmarm/queue' into devel-hourly-2014082810 git bisect good 5cde56d94f0761101b235d1c88c674034bdd101a # 11:23 28+ 1 Merge 'netdev-next/master' into devel-hourly-2014082810 git bisect bad dd7ed2643b17336d0103ddda2c385b59812a40b8 # 11:26 5- 14 Merge 'hid/for-3.18/picolcd' into devel-hourly-2014082810 git bisect bad aa6fe56cf691d0587229cb5c21c44899fc312134 # 11:28 2- 6 Merge 'rcu/rcu/exp' into devel-hourly-2014082810 git bisect bad a31a69e0a7601d78b242d1e0c367e53e6bdc47f8 # 11:31 1- 3 rcu: Eliminate deadlock between CPU hotplug and expedited grace periods git bisect bad 86d8f10e35c1e63d5a839792efd7c3cb6a564fe4 # 11:34 4- 6 rcutorture: Add preemption-slam testing # first bad commit: [86d8f10e35c1e63d5a839792efd7c3cb6a564fe4] rcutorture: Add preemption-slam testing git bisect good b52938be99a099b452979ed805b480337ee2df0f # 11:41 120+ 8 Merge branch 'rcu-tasks.2014.08.27a' into HEAD git bisect bad 8153a6bbcc340b292248b66a6db89b991fe23995 # 11:41 0- 11 0day head guard for 'devel-hourly-2014082810' git bisect good f1bd473f95e02bc382d4dae94d7f82e2a455e05d # 12:00 120+ 3 Merge branch 'sec-v3.17-rc2' of git://git.kernel.org/pub/scm/linux/kernel/git/sergeh/linux-security git bisect good d05446ae2128064a4bb8f74c84f6901ffb5c94bc # 12:24 120+ 4 Add linux-next specific files for 20140827 This script may reproduce the error. ---------------------------------------------------------------------------- #!/bin/bash kernel=$1 initrd=quantal-core-x86_64.cgz wget --no-clobber https://github.com/fengguang/reproduce-kernel-bug/raw/master/initrd/$initrd kvm=( qemu-system-x86_64 -enable-kvm -cpu Haswell,+smep,+smap -kernel $kernel -initrd $initrd -m 320 -smp 2 -net nic,vlan=1,model=e1000 -net user,vlan=1 -boot order=nc -no-reboot -watchdog i6300esb -rtc base=localtime -serial stdio -display none -monitor null ) append=( hung_task_panic=1 earlyprintk=ttyS0,115200 debug apic=debug sysrq_always_enabled rcupdate.rcu_cpu_stall_timeout=100 panic=-1 softlockup_panic=1 nmi_watchdog=panic oops=panic load_ramdisk=2 prompt_ramdisk=0 console=ttyS0,115200 console=tty0 vga=normal root=/dev/ram0 rw drbd.minor_count=8 ) "${kvm[@]}" --append "${append[*]}" ---------------------------------------------------------------------------- Thanks, Fengguang