Greetings,

I got the below dmesg and the first bad commit is

git://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git master

commit c7d2e322dd6eac1e3b44d2ea13f9818ad0f903de
Author:     Fabian Frederick <fabf@skynet.be>
AuthorDate: Sat Mar 22 12:49:51 2014 +1100
Commit:     Stephen Rothwell <sfr@canb.auug.org.au>
CommitDate: Mon Mar 24 17:32:07 2014 +1100

    ufs: sb mutex merge + mutex_destroy
    
    788257d6101d9 ("ufs: remove the BKL") replaced BKL with mutex protection
    using functions lock_ufs, unlock_ufs and struct mutex 'mutex' in sb_info.
    
    b6963327e052 ("ufs: drop lock/unlock super") removed lock/unlock super and
    added struct mutex 's_lock' in sb_info.
    
    Those 2 mutexes are generally locked/unlocked at the same time except in
    allocation (balloc, ialloc).
    
    This patch merges the 2 mutexes and propagates first commit solution.  It
    also adds mutex destruction before kfree during ufs_fill_super failure and
    ufs_put_super.
    
    Signed-off-by: Fabian Frederick <fabf@skynet.be>
    Cc: Evgeniy Dushistov <dushistov@mail.ru>
    Signed-off-by: Andrew Morton <akpm@linux-foundation.org>

+-----------------------------------------------------------------+------------+---------------+
|                                                                 | c7d2e322dd | next-20140324 |
+-----------------------------------------------------------------+------------+---------------+
| boot_successes                                                  | 31         | 4             |
| boot_failures                                                   | 26         | 15            |
| WARNING:CPU:PID:at_kernel/locking/mutex-debug.c:mutex_destroy() | 26         | 15            |
| INFO:rcu_preempt_detected_stalls_on_CPUs/tasks                  | 25         | 15            |
| INFO:task_blocked_for_more_than_seconds                         | 26         | 15            |
| RIP:flat_send_IPI_mask                                          | 26         | 15            |
| Kernel_panic-not_syncing:hung_task:blocked_tasks                | 26         | 15            |
| backtrace:do_mount                                              | 26         | 15            |
| backtrace:SyS_mount                                             | 26         | 15            |
| backtrace:user_path_create                                      | 25         | 15            |
| backtrace:SyS_mknodat                                           | 25         | 15            |
| backtrace:SyS_mknod                                             | 25         | 15            |
| backtrace:watchdog                                              | 26         | 15            |
| backtrace:work_fn                                               | 1          |               |
+-----------------------------------------------------------------+------------+---------------+

[    7.661278] UFSD (fs/ufs/super.c, 1264): ufs_fill_super:
[    7.664618] ufs was compiled with read-only support, can't be mounted as read-write
[    7.667790] UFSD (fs/ufs/super.c, 1264): ufs_fill_super:
[    7.670703] WARNING: CPU: 0 PID: 483 at kernel/locking/mutex-debug.c:117 mutex_destroy+0x4c/0x65()
[    7.674108] ufs was compiled with read-only support, can't be mounted as read-write
[    7.676652] flag 0
[    7.677363] ufs was compiled with read-only support, can't be mounted as read-write

git bisect start 098bc161bfc7697f7e2ca15400f13ca1ac60e9dd 9b28ec4b41883a83c3b0960e61739162443c5bba --
git bisect  bad 46a1de3d4918d4210affec733d665fd0ff2391e3  # 20:06     15-      4  Merge branch 'akpm/master'
git bisect good c51fabb1e3b203382616ca21f7cf4e7b0912d50e  # 20:20     57+      1  Revert "vmstat: use raw_cpu_ops to avoid false positives on preemption checks"
git bisect good bc82fcdfe572cea8d21ad97f1fab2d8cf3d412bc  # 20:45     57+      0  net: replace __this_cpu_inc in route.c with raw_cpu_inc
git bisect good 855370bf10fd8eb8877e197b6cbcd0262a9b34b7  # 21:04     57+      0  arm64: initialize pgprot info earlier in boot
git bisect good a5a8c89e515b8617f232d1a3375a086c4bfcae24  # 21:28     57+      0  fs/ufs: remove unused ufs_super_block_second pointer
git bisect  bad c7d2e322dd6eac1e3b44d2ea13f9818ad0f903de  # 21:48     22-      5  ufs: sb mutex merge + mutex_destroy
git bisect good e68aadb9f813be874722b4ef1e84b32557688364  # 22:11     57+      0  fs/ufs: remove unused ufs_super_block_third pointer
# first bad commit: [c7d2e322dd6eac1e3b44d2ea13f9818ad0f903de] ufs: sb mutex merge + mutex_destroy
git bisect good e68aadb9f813be874722b4ef1e84b32557688364  # 22:21    171+      0  fs/ufs: remove unused ufs_super_block_third pointer
git bisect  bad 098bc161bfc7697f7e2ca15400f13ca1ac60e9dd  # 22:21      0-     15  Add linux-next specific files for 20140324
git bisect good b098d6726bbfb94c06d6e1097466187afddae61f  # 23:12    171+      2  Linux 3.14-rc8
git bisect  bad 2ec6ab7445b659589abf5b9cf6c087c420d439bd  # 23:22     50-      1  Add linux-next specific files for 20140325


This script may reproduce the error.

-----------------------------------------------------------------------------
#!/bin/bash

kernel=$1
initrd=yocto-minimal-x86_64.cgz

wget --no-clobber https://github.com/fengguang/reproduce-kernel-bug/blob/master/initrd/$initrd

kvm=(
	qemu-system-x86_64 -cpu kvm64 -enable-kvm
	-kernel $kernel
	-initrd $initrd
	-smp 2
	-m 256M
	-net nic,vlan=0,macaddr=00:00:00:00:00:00,model=virtio
	-net user,vlan=0
	-net nic,vlan=1,model=e1000
	-net user,vlan=1
	-boot order=nc
	-no-reboot
	-watchdog i6300esb
	-serial stdio
	-display none
	-monitor null
)

append=(
	debug
	sched_debug
	apic=debug
	ignore_loglevel
	sysrq_always_enabled
	panic=10
	prompt_ramdisk=0
	earlyprintk=ttyS0,115200
	console=ttyS0,115200
	console=tty0
	vga=normal
	root=/dev/ram0
	rw
)

"${kvm[@]}" --append "${append[*]}"
-----------------------------------------------------------------------------

Thanks,
Fengguang