From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from aquinas.techsquare.com ([75.125.237.226]:38097 "EHLO techsquare.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1751915AbdA1Uv0 (ORCPT ); Sat, 28 Jan 2017 15:51:26 -0500 To: bo.li.liu@oracle.com References: <83c1a49a-b2ea-c7b6-151d-cbd15b6c9c93@techsquare.com> <20170125210618.GA18416@localhost.localdomain> Cc: linux-btrfs@vger.kernel.org From: Matt McKinnon Message-ID: <1d4c2360-0e6d-b476-e1af-2789ca01e59f@techsquare.com> Date: Sat, 28 Jan 2017 15:50:38 -0500 MIME-Version: 1.0 In-Reply-To: <20170125210618.GA18416@localhost.localdomain> Content-Type: text/plain; charset=windows-1252; format=flowed Subject: Re: Hard crash on 4.9.5 Sender: linux-btrfs-owner@vger.kernel.org List-ID: This same file system (which crashed again with the same errors) is also giving this output during a metadata or data balance: Jan 27 19:42:47 my_machine kernel: [ 335.018123] BTRFS info (device sda1): no csum found for inode 28472371 start 2191360 Jan 27 19:42:47 my_machine kernel: [ 335.018128] BTRFS info (device sda1): no csum found for inode 28472371 start 2195456 Jan 27 19:42:47 my_machine kernel: [ 335.018491] BTRFS info (device sda1): no csum found for inode 28472371 start 4018176 Jan 27 19:42:47 my_machine kernel: [ 335.018496] BTRFS info (device sda1): no csum found for inode 28472371 start 4022272 Jan 27 19:42:47 my_machine kernel: [ 335.018499] BTRFS info (device sda1): no csum found for inode 28472371 start 4026368 Jan 27 19:42:47 my_machine kernel: [ 335.018502] BTRFS info (device sda1): no csum found for inode 28472371 start 4030464 Jan 27 19:42:47 my_machine kernel: [ 335.019443] BTRFS info (device sda1): no csum found for inode 28472371 start 6156288 Jan 27 19:42:47 my_machine kernel: [ 335.019688] BTRFS info (device sda1): no csum found for inode 28472371 start 7933952 Jan 27 19:42:47 my_machine kernel: [ 335.019693] BTRFS info (device sda1): no csum found for inode 28472371 start 7938048 Jan 27 19:42:47 my_machine kernel: [ 335.019754] BTRFS info (device sda1): no csum found for inode 28472371 start 8077312 Jan 27 19:42:47 my_machine kernel: [ 335.025485] BTRFS warning (device sda1): csum failed ino 28472371 off 2191360 csum 4031061501 expected csum 0 Jan 27 19:42:47 my_machine kernel: [ 335.025490] BTRFS warning (device sda1): csum failed ino 28472371 off 2195456 csum 2371784003 expected csum 0 Jan 27 19:42:47 my_machine kernel: [ 335.025526] BTRFS warning (device sda1): csum failed ino 28472371 off 4018176 csum 3812080098 expected csum 0 Jan 27 19:42:47 my_machine kernel: [ 335.025531] BTRFS warning (device sda1): csum failed ino 28472371 off 4022272 csum 2776681411 expected csum 0 Jan 27 19:42:47 my_machine kernel: [ 335.025534] BTRFS warning (device sda1): csum failed ino 28472371 off 4026368 csum 1179241675 expected csum 0 Jan 27 19:42:47 my_machine kernel: [ 335.025540] BTRFS warning (device sda1): csum failed ino 28472371 off 4030464 csum 1256914217 expected csum 0 Jan 27 19:42:47 my_machine kernel: [ 335.026142] BTRFS warning (device sda1): csum failed ino 28472371 off 7933952 csum 2695958066 expected csum 0 Jan 27 19:42:47 my_machine kernel: [ 335.026147] BTRFS warning (device sda1): csum failed ino 28472371 off 7938048 csum 3260800596 expected csum 0 Jan 27 19:42:47 my_machine kernel: [ 335.026934] BTRFS warning (device sda1): csum failed ino 28472371 off 6156288 csum 4293116449 expected csum 0 Jan 27 19:42:47 my_machine kernel: [ 335.033249] BTRFS warning (device sda1): csum failed ino 28472371 off 8077312 csum 4031878292 expected csum 0 Can these be ignored? On 01/25/2017 04:06 PM, Liu Bo wrote: > On Mon, Jan 23, 2017 at 03:03:55PM -0500, Matt McKinnon wrote: >> Wondering what to do about this error which says 'reboot needed'. Has >> happened a three times in the past week: >> > > Well, I don't think btrfs's logic here is wrong, the following stack > shows that a nfs client has sent a second unlink against the same inode > while somehow the inode was not fully deleted by the first unlink. > > So it'd be good that you could add some debugging information to get us > further. > > Thanks, > > -liubo > >> Jan 23 14:16:17 my_machine kernel: [ 2568.595648] BTRFS error (device sda1): >> err add delayed dir index item(index: 23810) into the deletion tree of the >> delayed node(root id: 257, inode id: 2661433, errno: -17) >> Jan 23 14:16:17 my_machine kernel: [ 2568.611010] ------------[ cut here >> ]------------ >> Jan 23 14:16:17 my_machine kernel: [ 2568.615628] kernel BUG at >> fs/btrfs/delayed-inode.c:1557! >> Jan 23 14:16:17 my_machine kernel: [ 2568.620942] invalid opcode: 0000 [#1] >> SMP >> Jan 23 14:16:17 my_machine kernel: [ 2568.624960] Modules linked in: ufs >> qnx4 hfsplus hfs minix ntfs msdos jfs xfs ipt_REJECT nf_rej >> ect_ipv4 xt_tcpudp nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack >> nf_conntrack iptable_filter ip_tables x_tables ipmi_devintf nfsd au >> th_rpcgss nfs_acl nfs lockd grace sunrpc fscache intel_rapl sb_edac >> edac_core x86_pkg_temp_thermal intel_powerclamp coretemp kvm_int >> el kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel >> aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper crypt >> d dm_multipath joydev mei_me mei lpc_ich ioatdma wmi ipmi_si ipmi_msghandler >> btrfs shpchp mac_hid lp parport ses enclosure scsi_tran >> sport_sas raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor >> async_tx xor raid6_pq libcrc32c igb hid_generic i2c_algo_ >> bit raid1 dca usbhid ahci raid0 ptp megaraid_sas multipath >> Jan 23 14:16:17 my_machine kernel: [ 2568.697150] hid libahci pps_core >> linear dm_mirror dm_region_hash dm_log >> Jan 23 14:16:17 my_machine kernel: [ 2568.702689] CPU: 0 PID: 2440 Comm: >> nfsd Tainted: G W 4.9.5-custom #1 >> Jan 23 14:16:17 my_machine kernel: [ 2568.710166] Hardware name: Supermicro >> X9DRH-7TF/7F/iTF/iF/X9DRH-7TF/7F/iTF/iF, BIOS 3.0b 04/28 >> /2014 >> Jan 23 14:16:17 my_machine kernel: [ 2568.719207] task: ffff95a42addab80 >> task.stack: ffffb9da85330000 >> Jan 23 14:16:17 my_machine kernel: [ 2568.725124] RIP: >> 0010:[] [] >> btrfs_delete_delayed_dir_inde >> x+0x286/0x290 [btrfs] >> Jan 23 14:16:17 my_machine kernel: [ 2568.735604] RSP: 0018:ffffb9da85333be0 >> EFLAGS: 00010286 >> Jan 23 14:16:17 my_machine kernel: [ 2568.740917] RAX: 0000000000000000 RBX: >> ffff95a3b104b690 RCX: 0000000000000000 >> Jan 23 14:16:17 my_machine kernel: [ 2568.748048] RDX: 0000000000000001 RSI: >> ffff95a42fc0dcc8 RDI: ffff95a42fc0dcc8 >> Jan 23 14:16:17 my_machine kernel: [ 2568.755171] RBP: ffffb9da85333c48 R08: >> 0000000000000491 R09: 0000000000000000 >> Jan 23 14:16:17 my_machine kernel: [ 2568.762297] R10: 0000000000000005 R11: >> 0000000000000006 R12: ffff95a3b104b6d8 >> Jan 23 14:16:17 my_machine kernel: [ 2568.769429] R13: 0000000000005d02 R14: >> ffff95a82953d800 R15: 00000000ffffffef >> Jan 23 14:16:17 my_machine kernel: [ 2568.776555] FS: 0000000000000000(0000) >> GS:ffff95a42fc00000(0000) knlGS:0000000000000000 >> Jan 23 14:16:17 my_machine kernel: [ 2568.784639] CS: 0010 DS: 0000 ES: >> 0000 CR0: 0000000080050033 >> Jan 23 14:16:17 my_machine kernel: [ 2568.790377] CR2: 00007f12ea376000 CR3: >> 00000003e1e07000 CR4: 00000000001406f0 >> Jan 23 14:16:17 my_machine kernel: [ 2568.797503] Stack: >> Jan 23 14:16:17 my_machine kernel: [ 2568.799524] ffffffff9b7fe5f2 >> ffff95a3b104b560 0000000000040000 ffff95a3f96b3e80 >> Jan 23 14:16:17 my_machine kernel: [ 2568.806983] ffff95a3f96b3e80 >> 39ff95a814eeeb68 600000000000289c 0000000000005d02 >> Jan 23 14:16:17 my_machine kernel: [ 2568.814436] ffff95a3f7457c40 >> ffff95a3bcb74138 ffff95a814eeeb68 0000000000289c39 >> Jan 23 14:16:17 my_machine kernel: [ 2568.821891] Call Trace: >> Jan 23 14:16:17 my_machine kernel: [ 2568.824343] [] ? >> mutex_lock+0x12/0x2f >> Jan 23 14:16:17 my_machine kernel: [ 2568.829671] [] >> __btrfs_unlink_inode+0x198/0x4c0 [btrfs] >> Jan 23 14:16:17 my_machine kernel: [ 2568.836555] [] >> btrfs_unlink_inode+0x1c/0x40 [btrfs] >> Jan 23 14:16:17 my_machine kernel: [ 2568.843086] [] >> btrfs_unlink+0x6b/0xb0 [btrfs] >> Jan 23 14:16:17 my_machine kernel: [ 2568.849091] [] >> vfs_unlink+0xda/0x190 >> Jan 23 14:16:17 my_machine kernel: [ 2568.854315] [] ? >> lookup_one_len+0xd3/0x130 >> Jan 23 14:16:17 my_machine kernel: [ 2568.860075] [] >> nfsd_unlink+0x16e/0x210 [nfsd] >> Jan 23 14:16:17 my_machine kernel: [ 2568.866084] [] >> nfsd3_proc_remove+0x7c/0x110 [nfsd] >> Jan 23 14:16:17 my_machine kernel: [ 2568.872529] [] >> nfsd_dispatch+0xb8/0x1f0 [nfsd] >> Jan 23 14:16:17 my_machine kernel: [ 2568.878641] [] >> svc_process_common+0x43f/0x700 [sunrpc] >> Jan 23 14:16:17 my_machine kernel: [ 2568.885432] [] >> svc_process+0xfc/0x1c0 [sunrpc] >> Jan 23 14:16:17 my_machine kernel: [ 2568.891528] [] >> nfsd+0xf0/0x160 [nfsd] >> Jan 23 14:16:17 my_machine kernel: [ 2568.896838] [] ? >> nfsd_destroy+0x60/0x60 [nfsd] >> Jan 23 14:16:17 my_machine kernel: [ 2568.902931] [] >> kthread+0xca/0xe0 >> Jan 23 14:16:17 my_machine kernel: [ 2568.907807] [] ? >> kthread_park+0x60/0x60 >> Jan 23 14:16:17 my_machine kernel: [ 2568.913296] [] >> ret_from_fork+0x25/0x30 >> Jan 23 14:16:17 my_machine kernel: [ 2568.918693] Code: ff ff 48 8b 43 10 49 >> 8b be f0 01 00 00 45 89 f9 4c 8b 03 4c 89 ea 48 c7 c6 f >> 0 8f 59 c0 48 8b 88 48 03 00 00 31 c0 e8 ba 36 f7 ff <0f> 0b 0f 1f 84 00 00 >> 00 00 00 0f 1f 44 00 00 55 48 89 e5 53 48 >> Jan 23 14:16:17 my_machine kernel: [ 2568.938651] RIP [] >> btrfs_delete_delayed_dir_index+0x286/0x290 [btrfs] >> Jan 23 14:16:17 my_machine kernel: [ 2568.946773] RSP >> Jan 23 14:16:17 my_machine kernel: [ 2568.996481] ---[ end trace >> e8c95b69e4ef5f70 ]--- >> Jan 23 14:16:19 my_machine kernel: [ 2570.503671] BUG: unable to handle >> kernel NULL pointer dereference at 0000000000000246 >> Jan 23 14:16:19 my_machine kernel: [ 2570.511551] IP: [] >> __wake_up_common+0x2b/0x90 >> Jan 23 14:16:19 my_machine kernel: [ 2570.517498] PGD 46a002067 >> Jan 23 14:16:19 my_machine kernel: [ 2570.520036] PUD 45af9c067 >> Jan 23 14:16:19 my_machine kernel: [ 2570.522748] PMD 0 >> Jan 23 14:16:19 my_machine kernel: [ 2570.523284] >> Jan 23 14:23:50 riperton kernel: [ 3021.853513] [] >> queued_spin_lock_slowpath+0xb/0xf >> Jan 23 14:23:50 riperton kernel: [ 3021.859776] [] >> _raw_spin_lock+0x20/0x30 >> Jan 23 14:23:50 riperton kernel: [ 3021.865261] [] >> pid_revalidate+0x4d/0xf0 >> Jan 23 14:23:50 riperton kernel: [ 3021.870747] [] >> lookup_fast+0x29b/0x2c0 >> Jan 23 14:23:50 riperton kernel: [ 3021.876147] [] >> path_openat+0x172/0x1370 >> Jan 23 14:16:19 my_machine kernel: [ 2570.524789] Oops: 0000 [#2] SMP >> Jan 23 14:16:19 my_machine kernel: [ 2570.527932] Modules linked in: ufs >> qnx4 hfsplus hfs minix ntfs msdos jfs xfs ipt_REJECT nf_reject_ipv4 >> xt_tcpudp nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack >> iptable_filter ip_tables x_tables ipmi_devintf nfsd auth_rpcgss nfs_acl nfs >> lockd grace sunrpc fscache intel_rapl sb_edac edac_core x86_pkg_temp_thermal >> intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul >> crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul >> glue_helper ablk_helper cryptd dm_multipath joydev mei_me mei lpc_ich >> ioatdma wmi ipmi_si ipmi_msghandler btrfs shpchp mac_hid lp parport ses >> enclosure scsi_transport_sas raid10 raid456 async_raid6_recov async_memcpy >> async_pq async_xor async_tx xor raid6_pq libcrc32c igb hid_generic >> i2c_algo_bit raid1 dca usbhid ahci raid0 ptp megaraid_sas multipath >> Jan 23 14:16:19 my_machine kernel: [ 2570.600135] hid libahci pps_core >> linear dm_mirror dm_region_hash dm_log >> Jan 23 14:16:19 my_machine kernel: [ 2570.605651] CPU: 2 PID: 2440 Comm: >> nfsd Tainted: G D W 4.9.5-custom #1 >> Jan 23 14:16:19 my_machine kernel: [ 2570.613128] Hardware name: Supermicro >> X9DRH-7TF/7F/iTF/iF/X9DRH-7TF/7F/iTF/iF, BIOS 3.0b 04/28/2014 >> Jan 23 14:16:19 my_machine kernel: [ 2570.622168] task: ffff95a42addab80 >> task.stack: ffffb9da85330000 >> Jan 23 14:16:19 my_machine kernel: [ 2570.628085] RIP: >> 0010:[] [] __wake_up_common+0x2b/0x90 >> Jan 23 14:16:19 my_machine kernel: [ 2570.636451] RSP: 0018:ffffb9da85333e58 >> EFLAGS: 00010082 >> Jan 23 14:16:19 my_machine kernel: [ 2570.641762] RAX: 0000000000000282 RBX: >> ffffb9da85333f18 RCX: 0000000000000000 >> Jan 23 14:16:19 my_machine kernel: [ 2570.648897] RDX: 0000000000000246 RSI: >> 0000000000000003 RDI: ffffb9da85333f18 >> Jan 23 14:16:19 my_machine kernel: [ 2570.656028] RBP: ffffb9da85333e90 R08: >> 0000000000000000 R09: ffff95a429c7ba00 >> Jan 23 14:16:19 my_machine kernel: [ 2570.663162] R10: 000002567df4f057 R11: >> 0000000000000001 R12: ffffb9da85333f20 >> Jan 23 14:16:19 my_machine kernel: [ 2570.670295] R13: 0000000000000282 R14: >> 0000000000000000 R15: 0000000000000003 >> Jan 23 14:16:19 my_machine kernel: [ 2570.677427] FS: 0000000000000000(0000) >> GS:ffff95a42fd00000(0000) knlGS:0000000000000000 >> Jan 23 14:16:19 my_machine kernel: [ 2570.685513] CS: 0010 DS: 0000 ES: >> 0000 CR0: 0000000080050033 >> Jan 23 14:16:19 my_machine kernel: [ 2570.691261] CR2: 0000000000000246 CR3: >> 000000045a400000 CR4: 00000000001406e0 >> Jan 23 14:16:19 my_machine kernel: [ 2570.698393] Stack: >> Jan 23 14:16:19 my_machine kernel: [ 2570.700411] 0000000100000246 >> 0000000000000000 ffffb9da85333f18 ffffb9da85333f10 >> Jan 23 14:16:19 my_machine kernel: [ 2570.707865] 0000000000000282 >> ffff95a42addab80 0000000000000000 ffffb9da85333ea0 >> Jan 23 14:16:19 my_machine kernel: [ 2570.715326] ffffffff9b0c0f43 >> ffffb9da85333ec8 ffffffff9b0c1967 ffff95a42addb2a8 >> Jan 23 14:16:19 my_machine kernel: [ 2570.722797] Call Trace: >> Jan 23 14:16:19 my_machine kernel: [ 2570.725267] [] >> __wake_up_locked+0x13/0x20 >> Jan 23 14:16:19 my_machine kernel: [ 2570.730923] [] >> complete+0x37/0x50 >> Jan 23 14:16:19 my_machine kernel: [ 2570.735892] [] >> mm_release+0xbf/0x140 >> Jan 23 14:16:19 my_machine kernel: [ 2570.741113] [] >> do_exit+0x13a/0xad0 >> Jan 23 14:16:19 my_machine kernel: [ 2570.746169] [] >> rewind_stack_do_exit+0x17/0x20 >> Jan 23 14:16:19 my_machine kernel: [ 2570.752170] Code: 0f 1f 44 00 00 55 48 >> 89 e5 41 57 41 89 f7 41 56 41 89 ce 41 55 41 54 4c 8d 67 08 53 48 83 ec 10 >> 89 55 cc 48 8b 57 08 4c 89 45 d0 <48> 8b 0a 49 39 d4 48 8d 42 e8 4c 8d 69 e8 >> 75 08 eb 38 4c 89 e8 >> Jan 23 14:16:19 my_machine kernel: [ 2570.772172] RIP [] >> __wake_up_common+0x2b/0x90 >> Jan 23 14:16:19 my_machine kernel: [ 2570.778196] RSP >> Jan 23 14:16:19 my_machine kernel: [ 2570.781680] CR2: 0000000000000246 >> Jan 23 14:16:19 my_machine kernel: [ 2570.784993] ---[ end trace >> e8c95b69e4ef5f71 ]--- >> Jan 23 14:16:19 my_machine kernel: [ 2570.794692] Fixing recursive fault but >> reboot is needed! >> -- >> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in >> the body of a message to majordomo@vger.kernel.org >> More majordomo info at http://vger.kernel.org/majordomo-info.html