Greetings, 0day kernel testing robot got the below dmesg and the first bad commit is git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git master commit 2c575026fae6e63771bd2a4c1d407214a8096a89 Author: Hong Zhiguo AuthorDate: Wed Nov 20 10:35:05 2013 -0700 Commit: Jens Axboe CommitDate: Wed Nov 20 15:33:04 2013 -0700 Update of blkg_stat and blkg_rwstat may happen in bh context. While u64_stats_fetch_retry is only preempt_disable on 32bit UP system. This is not enough to avoid preemption by bh and may read strange 64 bit value. Signed-off-by: Hong Zhiguo Acked-by: Tejun Heo Cc: stable(a)kernel.org Signed-off-by: Jens Axboe +-----------------------------------------------------------------------------+------------+------------+------------------+ | | 82023bb7f7 | 2c575026fa | v3.16-rc7_080120 | +-----------------------------------------------------------------------------+------------+------------+------------------+ | boot_successes | 270 | 22 | 7 | | boot_failures | 0 | 68 | 4 | | BUG:unable_to_handle_kernel_NULL_pointer_dereference | 0 | 1 | | | Oops | 0 | 1 | | | EIP_is_at_strlen | 0 | 1 | | | BUG:sleeping_function_called_from_invalid_context_at_kernel/locking/rwsem.c | 0 | 1 | | | INFO:lockdep_is_turned_off | 0 | 1 | | | backtrace:vfs_write | 0 | 1 | | | backtrace:SyS_write | 0 | 1 | | | WARNING:CPU:PID:at_kernel/softirq.c:_local_bh_enable_ip() | 0 | 67 | | | inconsistent_SOFTIRQ-ON-W-IN-SOFTIRQ-W_usage | 0 | 67 | 4 | | backtrace:redo_fd_request | 0 | 67 | | | backtrace:floppy_work_workfn | 0 | 0 | 4 | | backtrace:cpu_startup_entry | 0 | 0 | 4 | +-----------------------------------------------------------------------------+------------+------------+------------------+ [ 2.848648] NX-protecting the kernel data: 5832k [ 2.903073] udev[122]: starting version 164 [ 3.123339] ------------[ cut here ]------------ [ 3.123386] WARNING: CPU: 0 PID: 85 at kernel/softirq.c:156 _local_bh_enable_ip+0x34/0x97() [ 3.123386] Modules linked in: [ 3.123386] CPU: 0 PID: 85 Comm: kworker/u2:2 Not tainted 3.12.0-10878-g2c57502 #1 [ 3.123386] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 [ 3.123386] Workqueue: floppy redo_fd_request [ 3.123386] c0258bca 00000000 d277bce8 c0646df5 d277bd00 c022d793 c0230f4d c04297b7 [ 3.123386] d2746634 d277bd6c d277bd10 c022d81d 00000009 00000000 d277bd1c c0230f4d [ 3.123386] d277bd4c d277bd24 c0230fc0 d277bd38 c04297b7 c001be10 d2746400 d2743c00 [ 3.123386] Call Trace: [ 3.123386] [] ? console_unlock+0x355/0x382 [ 3.123386] [] dump_stack+0x16/0x18 [ 3.123386] [] warn_slowpath_common+0x5f/0x76 [ 3.123386] [] ? _local_bh_enable_ip+0x34/0x97 [ 3.123386] [] ? blkg_rwstat_read+0x1d/0x24 [ 3.123386] [] warn_slowpath_null+0x14/0x18 [ 3.123386] [] _local_bh_enable_ip+0x34/0x97 [ 3.123386] [] local_bh_enable+0x10/0x12 [ 3.123386] [] blkg_rwstat_read+0x1d/0x24 [ 3.123386] [] __cfq_set_active_queue+0x65/0x165 [ 3.123386] [] ? mark_lock+0x1e/0x1d3 [ 3.123386] [] ? mark_lock+0x1e/0x1d3 [ 3.123386] [] ? __lock_is_held+0x2d/0x43 [ 3.123386] [] ? pvclock_clocksource_read+0xb1/0xda [ 3.123386] [] ? kvm_clock_read+0x14/0x1d [ 3.123386] [] ? paravirt_sched_clock+0x9/0xd [ 3.123386] [] cfq_dispatch_requests+0x521/0x78a [ 3.123386] [] ? __lock_acquire+0x523/0xbdf [ 3.123386] [] ? __lock_acquire+0x523/0xbdf [ 3.123386] [] ? mark_lock+0x1e/0x1d3 [ 3.123386] [] ? lock_acquire+0xa3/0xfc [ 3.123386] [] ? redo_fd_request+0x3f/0xc0c [ 3.123386] [] blk_peek_request+0x59/0x211 [ 3.123386] [] blk_fetch_request+0xe/0x1e [ 3.123386] [] redo_fd_request+0x7a/0xc0c [ 3.123386] [] ? process_one_work+0x13c/0x398 [ 3.123386] [] process_one_work+0x1d4/0x398 [ 3.123386] [] worker_thread+0x12b/0x1c8 [ 3.123386] [] ? rescuer_thread+0x1ff/0x1ff [ 3.123386] [] kthread+0x8b/0x90 [ 3.123386] [] ret_from_kernel_thread+0x1b/0x28 [ 3.123386] [] ? insert_kthread_work+0x5a/0x5a [ 3.123386] ---[ end trace 470268993472ede8 ]--- [ 3.246919] end_request: I/O error, dev fd0, sector 0 git bisect start v3.13 v3.12 -- git bisect good 3bad8bb5cd3048a67df43ac6b1e2f191f19d9ff0 # 22:45 62+ 0 Merge branch 'for-next' of git://git.samba.org/sfrench/cifs-2.6 git bisect bad dd0508093b79141e0044ca02f0acb6319f69f546 # 22:59 0- 62 Merge branch 'sched-urgent-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip git bisect bad 3f02ff5c2c69753666787ed125708d283a823ffb # 23:11 5- 26 Merge tag 'tty-3.13-rc3' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/tty git bisect good 1ab231b274ba51a54acebec23c6aded0f3cdf54e # 23:19 62+ 5 Merge branch 'timers-urgent-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip git bisect bad 5ee540613db504a10e15fafaf4c08cac96aa1823 # 23:32 7- 5 Merge branch 'for-linus' of git://git.kernel.dk/linux-block git bisect good 53c6de50262a8edd6932bb59a32db7b9d92f8d67 # 23:39 90+ 0 Merge branch 'x86/urgent' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip git bisect good 59fb2f0e9e30ad99a8bab0ff1efaf8f4a3b7105f # 00:12 90+ 0 Merge tag 'fbdev-fixes-3.13' of git://git.kernel.org/pub/scm/linux/kernel/git/tomba/linux git bisect good ef1e4e32d595d3e6c9a6d3d2956f087d5886c5e5 # 01:00 90+ 0 Merge tag 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/mason/linux-btrfs git bisect good 29be6345bbaec8502a70c4e2204d5818b48c4e8f # 01:08 90+ 0 Merge tag 'nfs-for-3.13-3' of git://git.linux-nfs.org/projects/trondmy/linux-nfs git bisect bad e345d767f6530ec9cb0aabab7ea248072a9c6975 # 01:38 31- 32 Merge branch 'stable/for-jens-3.13-take-two' of git://git.kernel.org/pub/scm/linux/kernel/git/xen/tip into for-linus git bisect bad c170bbb45febc03ac4d34ba2b8bb55e06104b7e7 # 01:43 28- 19 block: submit_bio_wait() conversions git bisect bad 2c575026fae6e63771bd2a4c1d407214a8096a89 # 02:04 6- 16 Update of blkg_stat and blkg_rwstat may happen in bh context. While u64_stats_fetch_retry is only preempt_disable on 32bit UP system. This is not enough to avoid preemption by bh and may read strange 64 bit value. # first bad commit: [2c575026fae6e63771bd2a4c1d407214a8096a89] Update of blkg_stat and blkg_rwstat may happen in bh context. While u64_stats_fetch_retry is only preempt_disable on 32bit UP system. This is not enough to avoid preemption by bh and may read strange 64 bit value. git bisect good 82023bb7f75b0052f40d3e74169d191c3e4e6286 # 02:29 270+ 0 Merge tag 'pm+acpi-2-3.13-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/rafael/linux-pm git bisect bad f1f29248d675d4762e7b7f498be50d2fb9c294f3 # 02:30 0- 4 0day head guard for 'devel-hourly-2014080120' git bisect bad 6f0928036bcbd8dd2ebee9201580da8bcc0a81af # 03:31 20- 20 Merge tag 'pm+acpi-3.16-rc8' of git://git.kernel.org/pub/scm/linux/kernel/git/rafael/linux-pm git bisect bad 4198cf324ddea447daac3f83d3c2319ced774201 # 03:38 16- 17 Add linux-next specific files for 20140801 This script may reproduce the error. ---------------------------------------------------------------------------- #!/bin/bash kernel=$1 initrd=yocto-minimal-i386.cgz wget --no-clobber https://github.com/fengguang/reproduce-kernel-bug/raw/master/initrd/$initrd kvm=( qemu-system-x86_64 -enable-kvm -cpu Haswell,+smep,+smap -kernel $kernel -initrd $initrd -m 320 -smp 1 -net nic,vlan=1,model=e1000 -net user,vlan=1 -boot order=nc -no-reboot -watchdog i6300esb -rtc base=localtime -serial stdio -display none -monitor null ) append=( hung_task_panic=1 earlyprintk=ttyS0,115200 debug apic=debug sysrq_always_enabled rcupdate.rcu_cpu_stall_timeout=100 panic=10 softlockup_panic=1 nmi_watchdog=panic prompt_ramdisk=0 console=ttyS0,115200 console=tty0 vga=normal root=/dev/ram0 rw drbd.minor_count=8 ) "${kvm[@]}" --append "${append[*]}" ---------------------------------------------------------------------------- Thanks, Fengguang _______________________________________________ LKP mailing list LKP(a)linux.intel.com