Hi Zhiguo, 0day kernel testing robot got the below dmesg and the first bad commit is git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git master commit 2c575026fae6e63771bd2a4c1d407214a8096a89 Author: Hong Zhiguo AuthorDate: Wed Nov 20 10:35:05 2013 -0700 Commit: Jens Axboe CommitDate: Wed Nov 20 15:33:04 2013 -0700 Update of blkg_stat and blkg_rwstat may happen in bh context. While u64_stats_fetch_retry is only preempt_disable on 32bit UP system. This is not enough to avoid preemption by bh and may read strange 64 bit value. Signed-off-by: Hong Zhiguo Acked-by: Tejun Heo Cc: stable@kernel.org Signed-off-by: Jens Axboe +-----------------------------------------------------------------------------+------------+------------+ | | 82023bb7f7 | 2c575026fa | +-----------------------------------------------------------------------------+------------+------------+ | boot_successes | 56 | 19 | | boot_failures | 73 | 24 | | Out_of_memory:Kill_process | 2 | 2 | | BUG:unable_to_handle_kernel_paging_request | 71 | 22 | | Oops | 71 | 22 | | EIP_is_at_process_one_work | 70 | 21 | | BUG:sleeping_function_called_from_invalid_context_at_kernel/locking/rwsem.c | 70 | 22 | | INFO:lockdep_is_turned_off | 70 | 22 | | EIP_is_at_kthread_data | 68 | 22 | | EIP_is_at_insert_work | 1 | | | Kernel_panic-not_syncing:Fatal_exception_in_interrupt | 1 | | | WARNING:CPU:PID:at_kernel/softirq.c:local_bh_enable() | 0 | 22 | | backtrace:do_mount | 0 | 19 | | backtrace:SyS_mount | 0 | 19 | | inconsistent_SOFTIRQ-ON-W-IN-SOFTIRQ-W_usage | 0 | 8 | | backtrace:redo_fd_request | 0 | 8 | | inconsistent_IN-SOFTIRQ-W-SOFTIRQ-ON-W_usage | 0 | 6 | | EIP_is_at_worker_thread | 0 | 1 | | backtrace:SyS_mprotect | 0 | 1 | | backtrace:error_code | 0 | 1 | | backtrace:smpboot_thread_fn | 0 | 1 | +-----------------------------------------------------------------------------+------------+------------+ it seems like your commit introduces "WARNING:CPU:PID:at_kernel/softirq.c:local_bh_enable()" oops messages. [ 8.933738] udev[102]: starting version 164 [ 10.273305] scsi_id (215) used greatest stack depth: 6328 bytes left [ 11.860972] ------------[ cut here ]------------ [ 11.862089] WARNING: CPU: 0 PID: 305 at kernel/softirq.c:156 local_bh_enable+0xa1/0x1a7() [ 11.864304] CPU: 0 PID: 305 Comm: mount Not tainted 3.12.0-10878-g2c57502 #1 [ 11.865769] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 [ 11.866917] 00000000 caa03b74 c1db7b69 caa03b8c c104fec1 c10588c1 c15aaf27 cc1fa908 [ 11.870084] cd25dc00 caa03b9c c105003b 00000009 00000000 caa03ba8 c10588c1 cd25be00 [ 11.870084] caa03c50 c15aaf27 00000000 00000000 00000001 00000000 cc1f002f cd25be00 [ 11.870084] Call Trace: [ 11.870084] [] dump_stack+0x40/0x5e [ 11.870084] [] warn_slowpath_common+0xae/0xef [ 11.870084] [] ? local_bh_enable+0xa1/0x1a7 [ 11.870084] [] ? __cfq_set_active_queue+0x1c9/0x359 [ 11.870084] [] warn_slowpath_null+0x30/0x42 [ 11.870084] [] local_bh_enable+0xa1/0x1a7 [ 11.870084] [] __cfq_set_active_queue+0x1c9/0x359 [ 11.870084] [] ? cfq_add_cfqq_rr+0x185/0x1ac [ 11.870084] [] ? cfq_update_idle_window+0x105/0x2ac [ 11.870084] [] ? mark_held_locks+0x163/0x199 [ 11.870084] [] ? _raw_spin_unlock_irq+0x5f/0x98 [ 11.870084] [] ? trace_hardirqs_on+0x27/0x37 [ 11.870084] [] ? _raw_spin_unlock_irq+0x5f/0x98 [ 11.870084] [] ? cfq_set_request+0x575/0x5e1 [ 11.870084] [] ? cfq_get_next_cfqg+0xe/0x12b [ 11.870084] [] cfq_dispatch_requests+0xbdb/0x12ce [ 11.870084] [] ? cfq_insert_request+0x3f7/0x87c [ 11.870084] [] ? __lock_acquire+0x14f6/0x1606 [ 11.870084] [] ? __lock_is_held+0x46/0x7b [ 11.870084] [] blk_peek_request+0xa3/0x522 [ 11.870084] [] blk_fetch_request+0x1c/0x56 [ 11.870084] [] do_nbd_request+0x23/0x1c9 [ 11.870084] [] __blk_run_queue+0x5a/0x71 [ 11.870084] [] blk_queue_bio+0x4cc/0x858 [ 11.870084] [] generic_make_request+0xdb/0x158 [ 11.870084] [] submit_bio+0xad/0x1d7 [ 11.870084] [] ? bio_init+0x9/0x35 [ 11.870084] [] ? bio_alloc_bioset+0x16f/0x275 [ 11.870084] [] _submit_bh+0x304/0x34a [ 11.870084] [] submit_bh+0x1d/0x2d [ 11.870084] [] __bread+0x11d/0x1c1 [ 11.870084] [] ext4_fill_super+0x354/0x446c [ 11.870084] [] ? strnlen+0x8/0x3a [ 11.870084] [] ? string.isra.3+0x45/0x11d [ 11.870084] [] ? snprintf+0x23/0x33 [ 11.870084] [] ? set_blocksize+0xe/0x15c [ 11.870084] [] ? disk_name+0x33/0xd7 [ 11.870084] [] mount_bdev+0x220/0x2e9 [ 11.870084] [] ? ext4_calculate_overhead+0x512/0x512 [ 11.870084] [] ? mount_bdev+0x220/0x2e9 [ 11.870084] [] ? ext4_calculate_overhead+0x512/0x512 [ 11.870084] [] ? __kmalloc_track_caller+0x260/0x4ec [ 11.870084] [] ? alloc_vfsmnt+0x19d/0x213 [ 11.870084] [] ext4_mount+0x23/0x33 [ 11.870084] [] ? ext4_calculate_overhead+0x512/0x512 [ 11.870084] [] mount_fs+0x21/0x149 [ 11.870084] [] vfs_kern_mount+0x93/0x1db [ 11.870084] [] do_mount+0x1167/0x1372 [ 11.870084] [] ? copy_mount_options+0xe/0x1a6 [ 11.870084] [] SyS_mount+0x101/0x184 [ 11.870084] [] syscall_call+0x7/0xb [ 11.870084] ---[ end trace 6f4587f6d881af3f ]--- [ 12.147799] block nbd15: Attempted send on closed socket Attached dmesg for the parent commit, too, to help confirm whether it is a noise error. git bisect start v3.13 v3.12 -- git bisect good 3bad8bb5cd3048a67df43ac6b1e2f191f19d9ff0 # 00:07 43+ 22 Merge branch 'for-next' of git://git.samba.org/sfrench/cifs-2.6 git bisect bad dd0508093b79141e0044ca02f0acb6319f69f546 # 00:29 1- 20 Merge branch 'sched-urgent-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip git bisect bad 3f02ff5c2c69753666787ed125708d283a823ffb # 00:40 0- 1 Merge tag 'tty-3.13-rc3' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/tty git bisect good 1ab231b274ba51a54acebec23c6aded0f3cdf54e # 00:50 43+ 27 Merge branch 'timers-urgent-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip git bisect bad 5ee540613db504a10e15fafaf4c08cac96aa1823 # 00:54 5- 14 Merge branch 'for-linus' of git://git.kernel.dk/linux-block git bisect good 53c6de50262a8edd6932bb59a32db7b9d92f8d67 # 01:02 43+ 20 Merge branch 'x86/urgent' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip git bisect good 59fb2f0e9e30ad99a8bab0ff1efaf8f4a3b7105f # 01:14 43+ 13 Merge tag 'fbdev-fixes-3.13' of git://git.kernel.org/pub/scm/linux/kernel/git/tomba/linux git bisect good ef1e4e32d595d3e6c9a6d3d2956f087d5886c5e5 # 01:31 43+ 26 Merge tag 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/mason/linux-btrfs git bisect good 29be6345bbaec8502a70c4e2204d5818b48c4e8f # 01:45 43+ 31 Merge tag 'nfs-for-3.13-3' of git://git.linux-nfs.org/projects/trondmy/linux-nfs git bisect bad e345d767f6530ec9cb0aabab7ea248072a9c6975 # 02:02 1- 25 Merge branch 'stable/for-jens-3.13-take-two' of git://git.kernel.org/pub/scm/linux/kernel/git/xen/tip into for-linus git bisect bad c170bbb45febc03ac4d34ba2b8bb55e06104b7e7 # 02:23 6- 23 block: submit_bio_wait() conversions git bisect bad 2c575026fae6e63771bd2a4c1d407214a8096a89 # 02:33 3- 10 Update of blkg_stat and blkg_rwstat may happen in bh context. While u64_stats_fetch_retry is only preempt_disable on 32bit UP system. This is not enough to avoid preemption by bh and may read strange 64 bit value. # first bad commit: [2c575026fae6e63771bd2a4c1d407214a8096a89] Update of blkg_stat and blkg_rwstat may happen in bh context. While u64_stats_fetch_retry is only preempt_disable on 32bit UP system. This is not enough to avoid preemption by bh and may read strange 64 bit value. git bisect good 82023bb7f75b0052f40d3e74169d191c3e4e6286 # 02:44 129+ 73 Merge tag 'pm+acpi-2-3.13-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/rafael/linux-pm git bisect bad b2fef0d0972b3fa998a96a5eb07904575b196738 # 02:44 0- 13 Add linux-next specific files for 20140515 git bisect bad 14186fea0cb06bc43181ce239efe0df6f1af260a # 02:50 80- 104 Merge tag 'locks-v3.15-4' of git://git.samba.org/jlayton/linux git bisect bad b2fef0d0972b3fa998a96a5eb07904575b196738 # 02:50 0- 13 Add linux-next specific files for 20140515 This script may reproduce the error. ----------------------------------------------------------------------------- #!/bin/bash kernel=$1 initrd=yocto-minimal-i386.cgz wget --no-clobber https://github.com/fengguang/reproduce-kernel-bug/blob/master/initrd/$initrd kvm=( qemu-system-x86_64 -cpu kvm64 -enable-kvm -kernel $kernel -initrd $initrd -smp 2 -m 256M -net nic,vlan=0,macaddr=00:00:00:00:00:00,model=virtio -net user,vlan=0 -net nic,vlan=1,model=e1000 -net user,vlan=1 -boot order=nc -no-reboot -watchdog i6300esb -serial stdio -display none -monitor null ) append=( debug sched_debug apic=debug ignore_loglevel sysrq_always_enabled panic=10 prompt_ramdisk=0 earlyprintk=ttyS0,115200 console=ttyS0,115200 console=tty0 vga=normal root=/dev/ram0 rw ) "${kvm[@]}" --append "${append[*]}" ----------------------------------------------------------------------------- Thanks, Jet