3.10 GA: ext4 crash on power7

[CAI Qian <caiqian@redhat.com>]

This has never been seen during either 3.10-rc7 or the last year's ext4
devel tree testing. Maybe because I have recently upgraded the distro
of the Fedora to the latest. This has happened using the fs_mark test and
reproduced on all power7 systems.

[ 2803.493590] EXT4-fs (ram0): mounted filesystem with ordered data mode. Opts: (null) 
[ 2806.684208] list_add corruption. next->prev should be prev (c00000002e7a2920), but was ffffffffffffffff. (next=c00000002291f6a8). 
[ 2806.684238] ------------[ cut here ]------------ 
[ 2806.684242] WARNING: at lib/list_debug.c:29 
[ 2806.684245] Modules linked in: vfat fat brd nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 nfs lockd sunrpc fscache nfnetlink_queue nfnetlink_log nfnetlink bluetooth rfkill arc4 md4 nls_utf8 cifs dns_resolver nf_tproxy_core nls_koi8_u nls_cp932 ts_kmp fuse nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_MASQUERADE ip6table_nat nf_nat_ipv6 ip6table_mangle ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 iptable_nat nf_nat_ipv4 nf_nat iptable_mangle ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables sg xfs libcrc32c ehea sd_mod crc_t10dif ibmvscsi scsi_transport_srp scsi_tgt dm_mirror dm_region_hash dm_log dm_mod [last unloaded: zlib] 
[ 2806.684339] CPU: 1 PID: 21270 Comm: fs_mark Not tainted 3.10.0 #1 
[ 2806.684344] task: c000000038ce0e10 ti: c0000000375c0000 task.ti: c0000000375c0000 
[ 2806.684348] NIP: c0000000003dcd00 LR: c0000000003dccfc CTR: 0000000001766760 
[ 2806.684352] REGS: c0000000375c31f0 TRAP: 0700   Not tainted  (3.10.0) 
[ 2806.684356] MSR: 8000000000029032 <SF,EE,ME,IR,DR,RI>  CR: 28002424  XER: 00000006 
[ 2806.684368] SOFTE: 1 
[ 2806.684370] CFAR: c00000000073c7e0 
[ 2806.684372]  
GPR00: c0000000003dccfc c0000000375c3470 c000000001107c10 0000000000000075  
GPR04: 0000000000000000 0000000000000000 c000000000af2ef8 c000000001522448  
GPR08: c000000000ae7c10 0000000000000000 0000000000000000 0000000000003fef  
GPR12: 0000000028002422 c00000000eda0400 0000000000000000 0000000000000000  
GPR16: 0000000000000000 0000000000000001 00007fffffffffff 0000000000000000  
GPR20: 0000000000000000 0000000000000001 0000000000000000 0000000000000010  
GPR24: 0000000000000000 0000000000000000 c0000000225d0028 c00000004a4f54f8  
GPR28: c000000049d99c00 c00000002291f6a8 c00000002e7a2920 c00000004a4f54f8  
[ 2806.684429] NIP [c0000000003dcd00] .__list_add+0x100/0x110 
[ 2806.684433] LR [c0000000003dccfc] .__list_add+0xfc/0x110 
[ 2806.684436] Call Trace: 
[ 2806.684440] [c0000000375c3470] [c0000000003dccfc] .__list_add+0xfc/0x110 (unreliable) 
[ 2806.684447] [c0000000375c3500] [c0000000002f395c] .ext4_mb_new_group_pa+0x22c/0x340 
[ 2806.684453] [c0000000375c35a0] [c0000000002fad34] .ext4_mb_new_blocks+0x634/0x690 
[ 2806.684458] [c0000000375c3670] [c0000000002ee3fc] .ext4_ext_map_blocks+0x6fc/0x1720 
[ 2806.684464] [c0000000375c37a0] [c0000000002bcfd4] .ext4_map_blocks+0x324/0x560 
[ 2806.684470] [c0000000375c3880] [c0000000002c03f0] .mpage_da_map_and_submit+0xb0/0x470 
[ 2806.684476] [c0000000375c3940] [c0000000002c0fbc] .ext4_da_writepages+0x32c/0x660 
[ 2806.684482] [c0000000375c3ad0] [c0000000001a27ac] .do_writepages+0x3c/0x70 
[ 2806.684487] [c0000000375c3b40] [c0000000001950b8] .__filemap_fdatawrite_range+0x68/0x80 
[ 2806.684493] [c0000000375c3be0] [c0000000001951fc] .filemap_write_and_wait_range+0x3c/0xb0 
[ 2806.684499] [c0000000375c3c70] [c0000000002b68f4] .ext4_sync_file+0x74/0x460 
[ 2806.684504] [c0000000375c3d20] [c00000000024d678] .do_fsync+0x78/0xd0 
[ 2806.684509] [c0000000375c3dc0] [c00000000024da78] .SyS_fsync+0x18/0x30 
[ 2806.684514] [c0000000375c3e30] [c000000000009e54] syscall_exit+0x0/0x98 
[ 2806.684518] Instruction dump: 
[ 2806.684521] 7fa4eb78 38634190 7fc6f378 4835fa89 60000000 0fe00000 4bffff54 3c62ff8b  
[ 2806.684530] 7fa6eb78 38634128 4835fa6d 60000000 <0fe00000> 4bffff2c 60000000 60000000  
[ 2806.684540] ---[ end trace a56068f561c7dbb3 ]--- 
[ 2806.741754] list_add corruption. next->prev should be prev (c00000002e7a29a0), but was ffffffffffffffff. (next=c00000002ec2fdf8). 
[ 2806.741782] ------------[ cut here ]------------ 
[ 2806.741786] WARNING: at lib/list_debug.c:29 
[ 2806.741788] Modules linked in: vfat fat brd nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 nfs lockd sunrpc fscache nfnetlink_queue nfnetlink_log nfnetlink bluetooth rfkill arc4 md4 nls_utf8 cifs dns_resolver nf_tproxy_core nls_koi8_u nls_cp932 ts_kmp fuse nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_MASQUERADE ip6table_nat nf_nat_ipv6 ip6table_mangle ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 iptable_nat nf_nat_ipv4 nf_nat iptable_mangle ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables sg xfs libcrc32c ehea sd_mod crc_t10dif ibmvscsi scsi_transport_srp scsi_tgt dm_mirror dm_region_hash dm_log dm_mod [last unloaded: zlib] 
[ 2806.741874] CPU: 0 PID: 21270 Comm: fs_mark Tainted: G        W    3.10.0 #1 
[ 2806.741879] task: c000000038ce0e10 ti: c0000000375c0000 task.ti: c0000000375c0000 
[ 2806.741883] NIP: c0000000003dcd00 LR: c0000000003dccfc CTR: 0000000001766760 
[ 2806.741887] REGS: c0000000375c31f0 TRAP: 0700   Tainted: G        W     (3.10.0) 
[ 2806.741890] MSR: 8000000000029032 <SF,EE,ME,IR,DR,RI>  CR: 28002424  XER: 00000006 
[ 2806.741901] SOFTE: 1 
[ 2806.741903] CFAR: c00000000073c7e0 
[ 2806.741906]  
GPR00: c0000000003dccfc c0000000375c3470 c000000001107c10 0000000000000075  
GPR04: 0000000000000000 0000000000000000 c000000000af2ef8 c000000001502448  
GPR08: c000000000ae7c10 0000000000000000 0000000000000000 0000000000003fef  
GPR12: 0000000028002422 c00000000eda0000 0000000000000000 0000000000000000  
GPR16: 0000000000000000 0000000000000001 00007fffffffffff 0000000000000000  
GPR20: 0000000000000000 0000000000000001 0000000000000000 0000000000000010  
GPR24: 0000000000000000 0000000000000000 c00000002647f7d0 c00000001b7b46c0  
GPR28: c000000049d99c00 c00000002ec2fdf8 c00000002e7a29a0 c00000001b7b46c0  
[ 2806.741957] NIP [c0000000003dcd00] .__list_add+0x100/0x110 
[ 2806.741961] LR [c0000000003dccfc] .__list_add+0xfc/0x110 
[ 2806.741964] Call Trace: 
[ 2806.741967] [c0000000375c3470] [c0000000003dccfc] .__list_add+0xfc/0x110 (unreliable) 
[ 2806.741974] [c0000000375c3500] [c0000000002f395c] .ext4_mb_new_group_pa+0x22c/0x340 
[ 2806.741980] [c0000000375c35a0] [c0000000002fad34] .ext4_mb_new_blocks+0x634/0x690 
[ 2806.741985] [c0000000375c3670] [c0000000002ee3fc] .ext4_ext_map_blocks+0x6fc/0x1720 
[ 2806.741991] [c0000000375c37a0] [c0000000002bcfd4] .ext4_map_blocks+0x324/0x560 
[ 2806.741996] [c0000000375c3880] [c0000000002c03f0] .mpage_da_map_and_submit+0xb0/0x470 
[ 2806.742001] [c0000000375c3940] [c0000000002c0fbc] .ext4_da_writepages+0x32c/0x660 
[ 2806.742007] [c0000000375c3ad0] [c0000000001a27ac] .do_writepages+0x3c/0x70 
[ 2806.742012] [c0000000375c3b40] [c0000000001950b8] .__filemap_fdatawrite_range+0x68/0x80 
[ 2806.742018] [c0000000375c3be0] [c0000000001951fc] .filemap_write_and_wait_range+0x3c/0xb0 
[ 2806.742023] [c0000000375c3c70] [c0000000002b68f4] .ext4_sync_file+0x74/0x460 
[ 2806.742027] [c0000000375c3d20] [c00000000024d678] .do_fsync+0x78/0xd0 
[ 2806.742032] [c0000000375c3dc0] [c00000000024da78] .SyS_fsync+0x18/0x30 
[ 2806.742038] [c0000000375c3e30] [c000000000009e54] syscall_exit+0x0/0x98 
[ 2806.742041] Instruction dump: 
[ 2806.742044] 7fa4eb78 38634190 7fc6f378 4835fa89 60000000 0fe00000 4bffff54 3c62ff8b  
[ 2806.742052] 7fa6eb78 38634128 4835fa6d 60000000 <0fe00000> 4bffff2c 60000000 60000000  
[ 2806.742061] ---[ end trace a56068f561c7dbb4 ]--- 
[ 2809.496203] list_add corruption. next->prev should be prev (c00000002e7a29a0), but was ffffffffffffffff. (next=c00000002ec2fdf8). 
[ 2809.496242] ------------[ cut here ]------------ 
[ 2809.496252] WARNING: at lib/list_debug.c:29 
[ 2809.496259] Modules linked in: vfat fat brd nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 nfs lockd sunrpc fscache nfnetlink_queue nfnetlink_log nfnetlink bluetooth rfkill arc4 md4 nls_utf8 cifs dns_resolver nf_tproxy_core nls_koi8_u nls_cp932 ts_kmp fuse nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_MASQUERADE ip6table_nat nf_nat_ipv6 ip6table_mangle ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 iptable_nat nf_nat_ipv4 nf_nat iptable_mangle ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables sg xfs libcrc32c ehea sd_mod crc_t10dif ibmvscsi scsi_transport_srp scsi_tgt dm_mirror dm_region_hash dm_log dm_mod [last unloaded: zlib] 
[ 2809.496367] CPU: 14 PID: 21348 Comm: fs_mark Tainted: G        W    3.10.0 #1 
[ 2809.496373] task: c000000038cdd6b0 ti: c0000000375b4000 task.ti: c0000000375b4000 
[ 2809.496378] NIP: c0000000003dcd00 LR: c0000000003dccfc CTR: 0000000001766760 
[ 2809.496383] REGS: c0000000375b71f0 TRAP: 0700   Tainted: G        W     (3.10.0) 
[ 2809.496388] MSR: 8000000000029032 <SF,EE,ME,IR,DR,RI>  CR: 28002424  XER: 00000006 
[ 2809.496401] SOFTE: 1 
[ 2809.496404] CFAR: c00000000073c7e0 
[ 2809.496407]  
GPR00: c0000000003dccfc c0000000375b7470 c000000001107c10 0000000000000075  
GPR04: 0000000000000000 0000000000000000 c000000000af2ef8 c0000000016c2448  
GPR08: c000000000ae7c10 0000000000000000 0000000000000000 0000000000003fef  
GPR12: 0000000028002422 c00000000eda3800 0000000000000000 0000000000000000  
GPR16: 0000000000000000 0000000000000001 00007fffffffffff 0000000000000000  
GPR20: 0000000000000000 0000000000000001 0000000000000000 0000000000000010  
GPR24: 0000000000000000 0000000000000000 c0000000227c0028 c00000004ae3a770  
GPR28: c000000049d99c00 c00000002ec2fdf8 c00000002e7a29a0 c00000004ae3a770  
[ 2809.496473] NIP [c0000000003dcd00] .__list_add+0x100/0x110 
[ 2809.496478] LR [c0000000003dccfc] .__list_add+0xfc/0x110 
[ 2809.496482] Call Trace: 
[ 2809.496486] [c0000000375b7470] [c0000000003dccfc] .__list_add+0xfc/0x110 (unreliable) 
[ 2809.496494] [c0000000375b7500] [c0000000002f395c] .ext4_mb_new_group_pa+0x22c/0x340 
[ 2809.496501] [c0000000375b75a0] [c0000000002fad34] .ext4_mb_new_blocks+0x634/0x690 
[ 2809.496507] [c0000000375b7670] [c0000000002ee3fc] .ext4_ext_map_blocks+0x6fc/0x1720 
[ 2809.496514] [c0000000375b77a0] [c0000000002bcfd4] .ext4_map_blocks+0x324/0x560 
[ 2809.496521] [c0000000375b7880] [c0000000002c03f0] .mpage_da_map_and_submit+0xb0/0x470 
[ 2809.496527] [c0000000375b7940] [c0000000002c0fbc] .ext4_da_writepages+0x32c/0x660 
[ 2809.496534] [c0000000375b7ad0] [c0000000001a27ac] .do_writepages+0x3c/0x70 
[ 2809.496541] [c0000000375b7b40] [c0000000001950b8] .__filemap_fdatawrite_range+0x68/0x80 
[ 2809.496547] [c0000000375b7be0] [c0000000001951fc] .filemap_write_and_wait_range+0x3c/0xb0 
[ 2809.496554] [c0000000375b7c70] [c0000000002b68f4] .ext4_sync_file+0x74/0x460 
[ 2809.496560] [c0000000375b7d20] [c00000000024d678] .do_fsync+0x78/0xd0 
[ 2809.496565] [c0000000375b7dc0] [c00000000024da78] .SyS_fsync+0x18/0x30 
[ 2809.496573] [c0000000375b7e30] [c000000000009e54] syscall_exit+0x0/0x98 
[ 2809.496577] Instruction dump: 
[ 2809.496581] 7fa4eb78 38634190 7fc6f378 4835fa89 60000000 0fe00000 4bffff54 3c62ff8b  
[ 2809.496591] 7fa6eb78 38634128 4835fa6d 60000000 <0fe00000> 4bffff2c 60000000 60000000  
[ 2809.496603] ---[ end trace a56068f561c7dbb5 ]--- 
2013-07-01 03:08:01,350 backend check_offset: INFO Rewriting file 1e7781a0-1329-4868-870f-3c066a988d49. 
[ 2814.504456] list_del corruption. next->prev should be c0000000305079f0, but was ffffffffffffffff 
[ 2814.504482] ------------[ cut here ]------------ 
[ 2814.504486] WARNING: at lib/list_debug.c:62 
[ 2814.504489] Modules linked in: vfat fat brd nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 nfs lockd sunrpc fscache nfnetlink_queue nfnetlink_log nfnetlink bluetooth rfkill arc4 md4 nls_utf8 cifs dns_resolver nf_tproxy_core nls_koi8_u nls_cp932 ts_kmp fuse nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_MASQUERADE ip6table_nat nf_nat_ipv6 ip6table_mangle ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 iptable_nat nf_nat_ipv4 nf_nat iptable_mangle ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables sg xfs libcrc32c ehea sd_mod crc_t10dif ibmvscsi scsi_transport_srp scsi_tgt dm_mirror dm_region_hash dm_log dm_mod [last unloaded: zlib] 
[ 2814.504584] CPU: 9 PID: 21470 Comm: umount Tainted: G        W    3.10.0 #1 
[ 2814.504589] task: c000000046367cd0 ti: c0000000469f0000 task.ti: c0000000469f0000 
[ 2814.504593] NIP: c0000000003dcda4 LR: c0000000003dcda0 CTR: 0000000001766760 
[ 2814.504598] REGS: c0000000469f3620 TRAP: 0700   Tainted: G        W     (3.10.0) 
[ 2814.504602] MSR: 8000000000029032 <SF,EE,ME,IR,DR,RI>  CR: 28000424  XER: 00000005 
[ 2814.504613] SOFTE: 1 
[ 2814.504616] CFAR: c00000000073c7e0 
[ 2814.504618]  
GPR00: c0000000003dcda0 c0000000469f38a0 c000000001107c10 0000000000000054  
GPR04: 0000000000000000 0000000000000000 c000000000af2ef8 c000000001622448  
GPR08: c000000000ae7c10 0000000000000000 0000000000000000 0000000000003fef  
GPR12: 0000000028000422 c00000000eda2400 0000000000000000 0000000000000000  
GPR16: 0000000000000000 0000000000000001 0000000000000008 ffffffffffffffff  
GPR20: c000000073eeea00 0000000000000020 0000000000000000 c00000000659b000  
GPR24: 0000000000000900 c00000002e7a2900 c000000049d99c00 0000000000000012  
GPR28: c000000001460840 c00000002e7a2920 c0000000305079e0 c0000000305079f0  
[ 2814.504675] NIP [c0000000003dcda4] .__list_del_entry+0x94/0xd0 
[ 2814.504680] LR [c0000000003dcda0] .__list_del_entry+0x90/0xd0 
[ 2814.504683] Call Trace: 
[ 2814.504686] [c0000000469f38a0] [c0000000003dcda0] .__list_del_entry+0x90/0xd0 (unreliable) 
[ 2814.504693] [c0000000469f3910] [c0000000003dcdf8] .list_del+0x18/0x50 
[ 2814.504699] [c0000000469f3990] [c0000000002f9f00] .ext4_mb_release+0x170/0x3e0 
[ 2814.504704] [c0000000469f3a80] [c0000000002de5f0] .ext4_put_super+0xb0/0x3d0 
[ 2814.504710] [c0000000469f3b20] [c000000000213320] .generic_shutdown_super+0x90/0x150 
[ 2814.504716] [c0000000469f3bb0] [c0000000002137a8] .kill_block_super+0x28/0x90 
[ 2814.504721] [c0000000469f3c30] [c000000000213c94] .deactivate_locked_super+0x94/0xd0 
[ 2814.504727] [c0000000469f3cb0] [c000000000239150] .mntput_no_expire+0x160/0x1d0 
[ 2814.504733] [c0000000469f3d50] [c00000000023a8ac] .SyS_umount+0xfc/0x4d0 
[ 2814.504738] [c0000000469f3e30] [c000000000009e54] syscall_exit+0x0/0x98 
[ 2814.504742] Instruction dump: 
[ 2814.504745] 4e800020 3c62ff8b 7d455378 38634210 4835f9e1 60000000 0fe00000 4bffffd8  
[ 2814.504755] 3c62ff8b 386342c0 4835f9c9 60000000 <0fe00000> 4bffffc0 3c62ff8b 38634280  
[ 2814.504764] ---[ end trace a56068f561c7dbb6 ]--- 
[ 2814.504777] Unable to handle kernel paging request for data at address 0xffffffffffffffff 
[ 2814.504781] Faulting instruction address: 0xc0000000003dcd48 
[ 2814.504786] Oops: Kernel access of bad area, sig: 11 [#1] 
[ 2814.504788] SMP NR_CPUS=1024 NUMA pSeries 
[ 2814.504793] Modules linked in: vfat fat brd nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 nfs lockd sunrpc fscache nfnetlink_queue nfnetlink_log nfnetlink bluetooth rfkill arc4 md4 nls_utf8 cifs dns_resolver nf_tproxy_core nls_koi8_u nls_cp932 ts_kmp fuse nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_MASQUERADE ip6table_nat nf_nat_ipv6 ip6table_mangle ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 iptable_nat nf_nat_ipv4 nf_nat iptable_mangle ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables sg xfs libcrc32c ehea sd_mod crc_t10dif ibmvscsi scsi_transport_srp scsi_tgt dm_mirror dm_region_hash dm_log dm_mod [last unloaded: zlib] 
[ 2814.504863] CPU: 9 PID: 21470 Comm: umount Tainted: G        W    3.10.0 #1 
[ 2814.504867] task: c000000046367cd0 ti: c0000000469f0000 task.ti: c0000000469f0000 
[ 2814.504871] NIP: c0000000003dcd48 LR: c0000000003dcdf8 CTR: 0000000001766760 
[ 2814.504876] REGS: c0000000469f3620 TRAP: 0300   Tainted: G        W     (3.10.0) 
[ 2814.504879] MSR: 8000000000009032 <SF,EE,ME,IR,DR,RI>  CR: 28000428  XER: 00000005 
[ 2814.504890] SOFTE: 1 
[ 2814.504892] CFAR: c00000000000908c 
[ 2814.504895] DAR: ffffffffffffffff, DSISR: 40000000 
[ 2814.504898]  
GPR00: c0000000003dcdf8 c0000000469f38a0 c000000001107c10 c00000002291f6a8  
GPR04: c00000002291f6a8 c0000000016362d0 0000000000000000 000000000009d606  
GPR08: 0000000000200200 ffffffffffffffff c0000000af3332c0 0000000000003fef  
GPR12: 0000000028000428 c00000000eda2400 0000000000000000 0000000000000000  
GPR16: 0000000000000000 0000000000000001 0000000000000008 ffffffffffffffff  
GPR20: c000000073eeea00 0000000000000020 0000000000000000 c00000000659b000  
GPR24: 0000000000000900 c00000002e7a2900 c000000049d99c00 0000000000000012  
GPR28: c000000001460840 c00000002e7a2920 c00000002291f698 c00000002291f6a8  
[ 2814.504953] NIP [c0000000003dcd48] .__list_del_entry+0x38/0xd0 
[ 2814.504957] LR [c0000000003dcdf8] .list_del+0x18/0x50 
[ 2814.504960] Call Trace: 
[ 2814.504964] [c0000000469f38a0] [0000000000000900] 0x900 (unreliable) 
[ 2814.504970] [c0000000469f3910] [c0000000003dcdf8] .list_del+0x18/0x50 
[ 2814.504975] [c0000000469f3990] [c0000000002f9f00] .ext4_mb_release+0x170/0x3e0 
[ 2814.504980] [c0000000469f3a80] [c0000000002de5f0] .ext4_put_super+0xb0/0x3d0 
[ 2814.504985] [c0000000469f3b20] [c000000000213320] .generic_shutdown_super+0x90/0x150 
[ 2814.504991] [c0000000469f3bb0] [c0000000002137a8] .kill_block_super+0x28/0x90 
[ 2814.504996] [c0000000469f3c30] [c000000000213c94] .deactivate_locked_super+0x94/0xd0 
[ 2814.505001] [c0000000469f3cb0] [c000000000239150] .mntput_no_expire+0x160/0x1d0 
[ 2814.505007] [c0000000469f3d50] [c00000000023a8ac] .SyS_umount+0xfc/0x4d0 
[ 2814.505012] [c0000000469f3e30] [c000000000009e54] syscall_exit+0x0/0x98 
[ 2814.505016] Instruction dump: 
[ 2814.505019] 61290100 7c641b78 f8010010 f821ff91 e9430000 7faa4800 e9230008 41de0044  
[ 2814.505028] 3d000020 61080200 7fa94000 41de0080 <e8a90000> 7fa32840 40de005c e8aa0008  
[ 2814.505040] ---[ end trace a56068f561c7dbb7 ]--- 
[ 2814.506104]  

Another system had trace like this,

[ 3037.222659] Unable to handle kernel paging request for data at address 0xffffffffffffffff 
[ 3037.222678] Faulting instruction address: 0xc0000000003dcd48 
[ 3037.222685] Oops: Kernel access of bad area, sig: 11 [#1] 
[ 3037.222690] SMP NR_CPUS=1024 NUMA pSeries 
[ 3037.222697] Modules linked in: vfat fat brd nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 nfs lockd sunrpc fscache nfnetlink_queue nfnetlink_log nfnetlink bluetooth rfkill arc4 md4 nls_utf8 cifs dns_resolver nf_tproxy_core nls_koi8_u nls_cp932 ts_kmp fuse nf_conntrack_netbios_ns nf_conntrack_broadcast ipt_MASQUERADE ip6table_nat nf_nat_ipv6 ip6table_mangle ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 iptable_nat nf_nat_ipv4 nf_nat iptable_mangle ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables sg xfs libcrc32c ehea ses enclosure sd_mod crc_t10dif ipr libata dm_mirror dm_region_hash dm_log dm_mod [last unloaded: zlib] 
[ 3037.222800] CPU: 5 PID: 18889 Comm: umount Not tainted 3.10.0 #1 
[ 3037.222806] task: c0000003c7453090 ti: c0000003c74d4000 task.ti: c0000003c74d4000 
[ 3037.222813] NIP: c0000000003dcd48 LR: c0000000003dcdf8 CTR: 0000000000000000 
[ 3037.222819] REGS: c0000003c74d7620 TRAP: 0300   Not tainted  (3.10.0) 
[ 3037.222824] MSR: 8000000000009032 <SF,EE,ME,IR,DR,RI>  CR: 24000428  XER: 00000000 
[ 3037.222840] SOFTE: 1 
[ 3037.222844] CFAR: c00000000000908c 
[ 3037.222848] DAR: ffffffffffffffff, DSISR: 40000000 
[ 3037.222852]  
GPR00: c0000000003dcdf8 c0000003c74d78a0 c000000001107c10 c0000003376232d8  
GPR04: c0000003376232d8 c000000354d91e00 c0000000002f9f50 0000000000000020  
GPR08: 0000000000200200 ffffffffffffffff c0000000062c28e8 0000000000000001  
GPR12: 0000000024000428 c00000000f241400 0000000000000000 0000000000000000  
GPR16: 0000000000000000 0000000000000001 0000000000000008 ffffffffffffffff  
GPR20: c00000035b0cbd00 0000000000000020 0000000000000000 c000000007162000  
GPR24: 0000000000000e80 c000000354d91e80 c0000003afb5a800 000000000000001d  
GPR28: c000000001460840 c000000354d91ea0 c0000003376232c8 c0000003376232d8  
[ 3037.222932] NIP [c0000000003dcd48] .__list_del_entry+0x38/0xd0 
[ 3037.222939] LR [c0000000003dcdf8] .list_del+0x18/0x50 
[ 3037.222943] Call Trace: 
[ 3037.222950] [c0000003c74d78a0] [0000000000000e00] 0xe00 (unreliable) 
[ 3037.222958] [c0000003c74d7910] [c0000000003dcdf8] .list_del+0x18/0x50 
[ 3037.222967] [c0000003c74d7990] [c0000000002f9f00] .ext4_mb_release+0x170/0x3e0 
[ 3037.222975] [c0000003c74d7a80] [c0000000002de5f0] .ext4_put_super+0xb0/0x3d0 
[ 3037.222983] [c0000003c74d7b20] [c000000000213320] .generic_shutdown_super+0x90/0x150 
[ 3037.222991] [c0000003c74d7bb0] [c0000000002137a8] .kill_block_super+0x28/0x90 
[ 3037.222999] [c0000003c74d7c30] [c000000000213c94] .deactivate_locked_super+0x94/0xd0 
[ 3037.223008] [c0000003c74d7cb0] [c000000000239150] .mntput_no_expire+0x160/0x1d0 
[ 3037.223016] [c0000003c74d7d50] [c00000000023a8ac] .SyS_umount+0xfc/0x4d0 
[ 3037.223025] [c0000003c74d7e30] [c000000000009e54] syscall_exit+0x0/0x98 
[ 3037.223030] Instruction dump: 
[ 3037.223035] 61290100 7c641b78 f8010010 f821ff91 e9430000 7faa4800 e9230008 41de0044  
[ 3037.223048] 3d000020 61080200 7fa94000 41de0080 <e8a90000> 7fa32840 40de005c e8aa0008  
[ 3037.223066] ---[ end trace b3d9173b8e607cb0 ]--- 
[ 3037.224148]  

Log indicated that crash happened while running this sub-test,
Running fs_mark test: ext4/4
***** Start of Random_MultiDir 2013.07.01 02:53:28 *****
./fs_mark -d /mnt/brd/testarea/testdir1 -d /mnt/brd/testarea/testdir2 -s 16384 -n 4096 -l /mnt/testarea/fs_4.log -r 32 -D 128

#  ./fs_mark  -d  /mnt/brd/testarea/testdir1  -d  /mnt/brd/testarea/testdir2  -s  16384  -n  4096  -l  /mnt/testarea/fs_4.log  -r  32  -D  128 
#	Version Version 3.2, 2 thread(s) starting at Mon Jul  1 02:53:28 2013
#	Sync method: INBAND FSYNC: fsync() per file in write loop.
#	Directories:  Time based hash between directories across 128 subdirectories with 180 seconds per subdirectory.
#	File names: 40 bytes long, (8 initial bytes of time stamp with 32 random bytes at end of name)
#	Files info: size 16384 bytes, written with an IO size of 16384 bytes per write
#	App overhead is time in microseconds spent in the inner file writing loop not doing file writing related system calls.

FSUse%        Count         Size    Files/sec     App Overhead
     8         8192        16384      10155.7            82026
Random_MultiDir Passed: 
 
***** End of Random_MultiDir 2013.07.01 02:53:30 *****
 
*******************************************************************************
\f


                     Number of $FSMARK_S_PARAM bytes Files/Sec

  10260 ++-----+-------+------+-------+------+-------+------+-------+-----++
  10250 ++     +       +      +       +      +      Two Directories + A   ++
  10230 ++                                                                ++
  10220 ++                                                                ++
  10210 ++                                                                ++
  10200 ++                                                                ++
  10180 ++                                                                ++
  10170 ++                                                                ++
  10160 ++                                A                               ++
  10150 ++                                                                ++
  10140 ++                                                                ++
  10120 ++                                                                ++
  10110 ++                                                                ++
  10100 ++                                                                ++
  10090 ++                                                                ++
  10070 ++                                                                ++
  10060 ++     +       +      +       +      +       +      +       +     ++
  10050 ++-----+-------+------+-------+------+-------+------+-------+-----++
       8100   8120    8140   8160    8180   8200    8220   8240    8260   8280
                                  Number of Files

*******************************************************************************

CAI Qian