* btrfs crash with a corrupted(?) filesystem
@ 2014-02-04 16:23 Roman Mamedov
2014-02-04 16:32 ` Hugo Mills
0 siblings, 1 reply; 4+ messages in thread
From: Roman Mamedov @ 2014-02-04 16:23 UTC (permalink / raw)
To: linux-btrfs
[-- Attachment #1: Type: text/plain, Size: 12223 bytes --]
Hello,
My server had a period of instability (PSU-related issues), some lockups,
some strange crashes, and some files became corrupted, and perhaps parts of
a filesystem too. One BTRFS partition now fails with the following errors.
On an attempt to make a snapshot:
[ 48.035664] btrfs: corrupt leaf, bad key order: block=193529446400,root=1, slot=9
[ 48.035795] ------------[ cut here ]------------
[ 48.035840] kernel BUG at fs/btrfs/inode.c:873!
[ 48.035884] invalid opcode: 0000 [#1] SMP
[ 48.036000] Modules linked in: cpufreq_stats cpufreq_userspace cpufreq_powersave cpufreq_conservative nfsd auth_rpcgss oid_registry nfs_acl nfs lockd dns_resolver fscache sunrpc 8021q garp mrp bridge stp llc fuse ext3 jbd it87 hwmon_vid ecryptfs snd_hda_codec_hdmi acpi_cpufreq snd_hda_codec_realtek mperf snd_hda_intel snd_hda_codec kvm_amd snd_pcsp snd_hwdep kvm fglrx(PO) snd_pcm_oss sp5100_tco snd_mixer_oss crc32c_intel ghash_clmulni_intel snd_pcm eeepc_wmi aesni_intel asus_wmi sparse_keymap uvcvideo videobuf2_vmalloc videobuf2_memops videobuf2_core psmouse rfkill snd_page_alloc video videodev edac_mce_amd snd_timer edac_core aes_x86_64 serio_raw evdev fam15h_power media snd ablk_helper joydev i2c_piix4 cp210x cryptd processor usbserial k10temp lrw i2c_core mxm_wmi thermal_sys gf128mul wmi glue_helper soundcore ehci_pci button ext4 crc16 jbd2 mbcache btrfs zlib_deflate crc32c libcrc32c dm_mod raid1 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq md_mod nbd sata_nv sg sd_mod crc_t10dif hid_generic usbhid hid sata_promise ohci_hcd ehci_hcd microcode xhci_hcd ahci ata_generic libahci libata usbcore r8169 mii usb_common scsi_mod
[ 48.040775] CPU: 4 PID: 4145 Comm: btrfs Tainted: P O 3.10.28-rm1+ #54
[ 48.040825] Hardware name: To be filled by O.E.M. To be filled by O.E.M./M5A97 LE R2.0, BIOS 1903 07/11/2013
[ 48.040876] task: ffff88040b4dc300 ti: ffff880409c14000 task.ti: ffff880409c14000
[ 48.040926] RIP: 0010:[<ffffffffa023f225>] [<ffffffffa023f225>] __cow_file_range+0x445/0x4e0 [btrfs]
[ 48.041042] RSP: 0018:ffff880409c15328 EFLAGS: 00010203
[ 48.041086] RAX: 0000000000001000 RBX: 0000000000000000 RCX: 0000000000000102
[ 48.041131] RDX: 0000000000000103 RSI: ffff88042e16a2e0 RDI: ffff88042e146e50
[ 48.041177] RBP: ffff880409c153e8 R08: 0000000000000000 R09: 000000000003ffff
[ 48.041222] R10: 0000000000040000 R11: 0000000000000001 R12: ffff8804352eb0a0
[ 48.041267] R13: 0000000000000000 R14: ffff88042e16a2e0 R15: 0000000000000000
[ 48.041318] FS: 00007f064ac9e780(0000) GS:ffff88044ed00000(0000) knlGS:0000000000000000
[ 48.041367] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 48.041411] CR2: 000000000046a9f0 CR3: 00000004388dc000 CR4: 00000000000407e0
[ 48.041456] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 48.041532] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 48.041576] Stack:
[ 48.041617] ffff880409c15388 ffff880430a2e800 ffff8804352eb130 ffff880409c15350
[ 48.041813] 000000000003ffff ffff88042e16a100 ffff880409c15350 ffffea000eb312c8
[ 48.042006] ffff88042e146e50 ff00000000ff16a0 ffff8804347c0800 0000000000000000
[ 48.042200] Call Trace:
[ 48.042274] [<ffffffffa0255a89>] ? free_extent_buffer+0x59/0xa0 [btrfs]
[ 48.042348] [<ffffffffa023f668>] run_delalloc_nocow+0x3a8/0xaf0 [btrfs]
[ 48.042421] [<ffffffffa02401c0>] run_delalloc_range+0x330/0x390 [btrfs]
[ 48.042495] [<ffffffffa0254641>] __extent_writepage+0x2f1/0x750 [btrfs]
[ 48.042570] [<ffffffffa0254d52>] extent_write_cache_pages.isra.31.constprop.47+0x2b2/0x3c0 [btrfs]
[ 48.042650] [<ffffffffa02550d7>] extent_writepages+0x47/0x60 [btrfs]
[ 48.042752] [<ffffffffa023bee0>] ? can_nocow_odirect+0x330/0x330 [btrfs]
[ 48.042823] [<ffffffffa0239873>] btrfs_writepages+0x23/0x30 [btrfs]
[ 48.042873] [<ffffffff8110ea19>] do_writepages+0x19/0x40
[ 48.042921] [<ffffffff81104711>] __filemap_fdatawrite_range+0x51/0x60
[ 48.042969] [<ffffffff811054ce>] filemap_fdatawrite_range+0xe/0x10
[ 48.043042] [<ffffffffa024f9e8>] btrfs_wait_ordered_range+0x48/0x110 [btrfs]
[ 48.043116] [<ffffffffa027499a>] __btrfs_write_out_cache+0x76a/0x990 [btrfs]
[ 48.043187] [<ffffffffa0232405>] ? btrfs_buffer_uptodate+0x65/0x80 [btrfs]
[ 48.043261] [<ffffffffa0274ec2>] btrfs_write_out_cache+0xb2/0xf0 [btrfs]
[ 48.045346] [<ffffffffa0255a89>] ? free_extent_buffer+0x59/0xa0 [btrfs]
[ 48.045414] [<ffffffffa0227383>] btrfs_write_dirty_block_groups+0x573/0x660 [btrfs]
[ 48.045489] [<ffffffffa02353a4>] commit_cowonly_roots+0x164/0x260 [btrfs]
[ 48.045560] [<ffffffffa023724c>] btrfs_commit_transaction+0x59c/0xab0 [btrfs]
[ 48.045614] [<ffffffff81075030>] ? finish_wait+0x80/0x80
[ 48.045686] [<ffffffffa026692a>] btrfs_mksubvol.isra.49+0x3aa/0x450 [btrfs]
[ 48.045759] [<ffffffffa0266aba>] btrfs_ioctl_snap_create_transid+0xea/0x170 [btrfs]
[ 48.045838] [<ffffffffa0266bfa>] ? btrfs_ioctl_snap_create_v2+0x3a/0x140 [btrfs]
[ 48.045917] [<ffffffffa0266cb9>] btrfs_ioctl_snap_create_v2+0xf9/0x140 [btrfs]
[ 48.045995] [<ffffffffa0268cc2>] btrfs_ioctl+0x942/0x1eb0 [btrfs]
[ 48.046043] [<ffffffff8112b111>] ? handle_mm_fault+0x251/0x350
[ 48.046091] [<ffffffff814a6bc4>] ? __do_page_fault+0x254/0x4f0
[ 48.046139] [<ffffffff81307287>] ? tty_ldisc_deref+0x37/0xa0
[ 48.046186] [<ffffffff8108604a>] ? __dequeue_entity+0x2a/0x50
[ 48.046232] [<ffffffff81173c77>] do_vfs_ioctl+0x87/0x560
[ 48.046279] [<ffffffff811741e1>] SyS_ioctl+0x91/0xb0
[ 48.046326] [<ffffffff814a9be9>] system_call_fastpath+0x16/0x1b
[ 48.046399] Code: 48 8b 7d 80 4c 89 f6 e8 fa 99 00 00 e9 75 fc ff ff 48 85 d2 74 50 80 be 30 fe ff ff 84 48 89 d1 74 44 48 ff c9 0f 84 26 fc ff ff <0f> 0b 48 8b 75 90 48 8b 7d 80 41 89 c0 b9 9b 03 00 00 48 c7 c2
[ 48.048869] RIP [<ffffffffa023f225>] __cow_file_range+0x445/0x4e0 [btrfs]
[ 48.048973] RSP <ffff880409c15328>
[ 48.049046] ---[ end trace 59138219c3d9423b ]---
Without trying to make that snapshot, but simply on an attempt to unmount:
[ 83.368744] btrfs: corrupt leaf, bad key order: block=193529446400,root=1, slot=9
[ 83.368876] ------------[ cut here ]------------
[ 83.368921] kernel BUG at fs/btrfs/inode.c:873!
[ 83.368965] invalid opcode: 0000 [#1] SMP
[ 83.369081] Modules linked in: nfsd auth_rpcgss oid_registry nfs_acl nfs lockd dns_resolver fscache sunrpc 8021q garp mrp bridge stp llc fuse ext3 jbd it87 hwmon_vid ecryptfs snd_hda_codec_hdmi fglrx(PO) snd_hda_codec_realtek snd_hda_intel snd_hda_codec snd_pcsp snd_hwdep snd_pcm_oss snd_mixer_oss snd_pcm acpi_cpufreq mperf eeepc_wmi kvm_amd kvm asus_wmi crc32c_intel psmouse ghash_clmulni_intel uvcvideo snd_page_alloc aesni_intel sparse_keymap serio_raw videobuf2_vmalloc videobuf2_memops videobuf2_core rfkill videodev fam15h_power joydev evdev aes_x86_64 ablk_helper video cryptd media sp5100_tco snd_timer lrw snd cp210x usbserial mxm_wmi processor gf128mul ehci_pci edac_mce_amd i2c_piix4 glue_helper soundcore k10temp button edac_core i2c_core thermal_sys wmi ext4 crc16 jbd2 mbcache btrfs zlib_deflate crc32c libcrc32c dm_mod raid1 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq md_mod nbd sata_nv sg sd_mod crc_t10dif hid_generic usbhid hid sata_promise microcode ahci libahci ata_generic ohci_hcd libata ehci_hcd scsi_mod xhci_hcd usbcore r8169 usb_common mii
[ 83.373704] CPU: 2 PID: 3024 Comm: umount Tainted: P O 3.10.28-rm1+ #54
[ 83.373753] Hardware name: To be filled by O.E.M. To be filled by O.E.M./M5A97 LE R2.0, BIOS 1903 07/11/2013
[ 83.373804] task: ffff8804386d8400 ti: ffff880439bac000 task.ti: ffff880439bac000
[ 83.373853] RIP: 0010:[<ffffffffa024b225>] [<ffffffffa024b225>] __cow_file_range+0x445/0x4e0 [btrfs]
[ 83.373970] RSP: 0018:ffff880439bad458 EFLAGS: 00010207
[ 83.374014] RAX: 0000000000001000 RBX: 0000000000000000 RCX: 0000000000000121
[ 83.374058] RDX: 0000000000000122 RSI: ffff88044dec6c10 RDI: ffff88044de6c9f0
[ 83.374103] RBP: ffff880439bad518 R08: 0000000000000000 R09: 000000000003ffff
[ 83.374177] R10: 0000000000040000 R11: 0000000000000001 R12: ffff88042f6c6f40
[ 83.374222] R13: 0000000000000000 R14: ffff88044dec6c10 R15: 0000000000000000
[ 83.374268] FS: 00007f166a7a7800(0000) GS:ffff88044ec80000(0000) knlGS:0000000000000000
[ 83.374317] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 83.374361] CR2: 00007f033d93648f CR3: 00000004390a9000 CR4: 00000000000407e0
[ 83.374406] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 83.374450] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 83.374494] Stack:
[ 83.374536] ffff880439bad4b8 ffff8804310d0800 ffff88042f6c6eb0 ffff880439bad480
[ 83.374729] 000000000003ffff ffff88044dec6a30 ffff880439bad480 ffffea000ea75640
[ 83.374922] ffff88044de6c9f0 ff00000000ff16a0 ffff880431097800 0000000000000000
[ 83.375115] Call Trace:
[ 83.375189] [<ffffffffa0261a89>] ? free_extent_buffer+0x59/0xa0 [btrfs]
[ 83.375261] [<ffffffffa024b668>] run_delalloc_nocow+0x3a8/0xaf0 [btrfs]
[ 83.375333] [<ffffffffa024c1c0>] run_delalloc_range+0x330/0x390 [btrfs]
[ 83.375436] [<ffffffffa0260641>] __extent_writepage+0x2f1/0x750 [btrfs]
[ 83.375510] [<ffffffffa0260d52>] extent_write_cache_pages.isra.31.constprop.47+0x2b2/0x3c0 [btrfs]
[ 83.377508] [<ffffffffa02610d7>] extent_writepages+0x47/0x60 [btrfs]
[ 83.377579] [<ffffffffa0247ee0>] ? can_nocow_odirect+0x330/0x330 [btrfs]
[ 83.377649] [<ffffffffa0245873>] btrfs_writepages+0x23/0x30 [btrfs]
[ 83.377699] [<ffffffff8110ea19>] do_writepages+0x19/0x40
[ 83.377747] [<ffffffff81104711>] __filemap_fdatawrite_range+0x51/0x60
[ 83.377824] [<ffffffff811054ce>] filemap_fdatawrite_range+0xe/0x10
[ 83.377897] [<ffffffffa025b9e8>] btrfs_wait_ordered_range+0x48/0x110 [btrfs]
[ 83.377971] [<ffffffffa028099a>] __btrfs_write_out_cache+0x76a/0x990 [btrfs]
[ 83.378040] [<ffffffffa023e405>] ? btrfs_buffer_uptodate+0x65/0x80 [btrfs]
[ 83.378114] [<ffffffffa0280ec2>] btrfs_write_out_cache+0xb2/0xf0 [btrfs]
[ 83.378188] [<ffffffffa0261a89>] ? free_extent_buffer+0x59/0xa0 [btrfs]
[ 83.378255] [<ffffffffa0233383>] btrfs_write_dirty_block_groups+0x573/0x660 [btrfs]
[ 83.378329] [<ffffffffa02413a4>] commit_cowonly_roots+0x164/0x260 [btrfs]
[ 83.378400] [<ffffffffa024324c>] btrfs_commit_transaction+0x59c/0xab0 [btrfs]
[ 83.378452] [<ffffffff81075030>] ? finish_wait+0x80/0x80
[ 83.378520] [<ffffffffa023e0eb>] btrfs_commit_super+0xbb/0x110 [btrfs]
[ 83.378589] [<ffffffffa023e2f8>] close_ctree+0x1b8/0x260 [btrfs]
[ 83.378637] [<ffffffff8117b609>] ? dispose_list+0x39/0x50
[ 83.378684] [<ffffffff8117c080>] ? evict_inodes+0xb0/0x110
[ 83.378753] [<ffffffffa0215a94>] btrfs_put_super+0x14/0x20 [btrfs]
[ 83.378801] [<ffffffff8116498c>] generic_shutdown_super+0x5c/0xe0
[ 83.378852] [<ffffffff81164aa1>] kill_anon_super+0x11/0x20
[ 83.378914] [<ffffffffa0217f15>] btrfs_kill_super+0x15/0x90 [btrfs]
[ 83.378963] [<ffffffff811652dd>] deactivate_locked_super+0x3d/0x70
[ 83.379040] [<ffffffff811658f9>] deactivate_super+0x49/0x70
[ 83.379086] [<ffffffff8117f6ec>] mntput_no_expire+0xfc/0x160
[ 83.379131] [<ffffffff81180777>] SyS_umount+0x97/0x3a0
[ 83.379178] [<ffffffff81064ef5>] ? sigprocmask+0x45/0x80
[ 83.379227] [<ffffffff814a9be9>] system_call_fastpath+0x16/0x1b
[ 83.379270] Code: 48 8b 7d 80 4c 89 f6 e8 fa 99 00 00 e9 75 fc ff ff 48 85 d2 74 50 80 be 30 fe ff ff 84 48 89 d1 74 44 48 ff c9 0f 84 26 fc ff ff <0f> 0b 48 8b 75 90 48 8b 7d 80 41 89 c0 b9 9b 03 00 00 48 c7 c2
[ 83.381744] RIP [<ffffffffa024b225>] __cow_file_range+0x445/0x4e0 [btrfs]
[ 83.381848] RSP <ffff880439bad458>
[ 83.381908] ---[ end trace 7161f57ab1b05921 ]---
Currently I have it mounted read-only, and all data seems to be accessible.
Short of copying everything away and recreating the FS, how can I bring it to
a working order. Is btrfsck a good option here?
Thanks
--
With respect,
Roman
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: btrfs crash with a corrupted(?) filesystem
2014-02-04 16:23 btrfs crash with a corrupted(?) filesystem Roman Mamedov
@ 2014-02-04 16:32 ` Hugo Mills
2014-02-04 16:35 ` Roman Mamedov
0 siblings, 1 reply; 4+ messages in thread
From: Hugo Mills @ 2014-02-04 16:32 UTC (permalink / raw)
To: Roman Mamedov; +Cc: linux-btrfs
[-- Attachment #1: Type: text/plain, Size: 1275 bytes --]
On Tue, Feb 04, 2014 at 10:23:10PM +0600, Roman Mamedov wrote:
> Hello,
>
> My server had a period of instability (PSU-related issues), some lockups,
> some strange crashes, and some files became corrupted, and perhaps parts of
> a filesystem too. One BTRFS partition now fails with the following errors.
>
> On an attempt to make a snapshot:
>
> [ 48.035664] btrfs: corrupt leaf, bad key order: block=193529446400,root=1, slot=9
[snip]
Bad key order is pretty much always down to hardware corrupting
data at some point -- which would go well with your list of hardware
problems above.
> Currently I have it mounted read-only, and all data seems to be accessible.
> Short of copying everything away and recreating the FS, how can I bring it to
> a working order. Is btrfsck a good option here?
The first investigation to do would be to look at the block in
question and see if it's got an obvious problem with it. If you post
the output of "btrfs-debug-tree -b 193529446400 /dev/whatever", we can
take a look at the indexing.
Hugo.
--
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
PGP key: 65E74AC0 from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
--- You're never alone with a rubber duck... ---
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: btrfs crash with a corrupted(?) filesystem
2014-02-04 16:32 ` Hugo Mills
@ 2014-02-04 16:35 ` Roman Mamedov
2014-02-04 16:40 ` Hugo Mills
0 siblings, 1 reply; 4+ messages in thread
From: Roman Mamedov @ 2014-02-04 16:35 UTC (permalink / raw)
To: Hugo Mills; +Cc: linux-btrfs
[-- Attachment #1: Type: text/plain, Size: 9196 bytes --]
On Tue, 4 Feb 2014 16:32:35 +0000
Hugo Mills <hugo@carfax.org.uk> wrote:
> On Tue, Feb 04, 2014 at 10:23:10PM +0600, Roman Mamedov wrote:
> > Hello,
> >
> > My server had a period of instability (PSU-related issues), some lockups,
> > some strange crashes, and some files became corrupted, and perhaps parts of
> > a filesystem too. One BTRFS partition now fails with the following errors.
> >
> > On an attempt to make a snapshot:
> >
> > [ 48.035664] btrfs: corrupt leaf, bad key order: block=193529446400,root=1, slot=9
> [snip]
>
> Bad key order is pretty much always down to hardware corrupting
> data at some point -- which would go well with your list of hardware
> problems above.
>
> > Currently I have it mounted read-only, and all data seems to be accessible.
> > Short of copying everything away and recreating the FS, how can I bring it to
> > a working order. Is btrfsck a good option here?
>
> The first investigation to do would be to look at the block in
> question and see if it's got an obvious problem with it. If you post
> the output of "btrfs-debug-tree -b 193529446400 /dev/whatever", we can
> take a look at the indexing.
Thanks; here it is:
# btrfs-debug-tree -b 193529446400 /dev/md4
leaf 193529446400 items 81 free space 46 generation 565572 owner 7
fs uuid dd12de99-bbe5-45cf-b869-6313c1f58431
chunk uuid b61f845a-ada5-4bcf-b995-7c5e1affa63d
item 0 key (EXTENT_CSUM EXTENT_CSUM 4278808576) itemoff 3955 itemsize 40
extent csum item
item 1 key (EXTENT_CSUM EXTENT_CSUM 4278853632) itemoff 3895 itemsize 60
extent csum item
item 2 key (EXTENT_CSUM EXTENT_CSUM 4278919168) itemoff 3883 itemsize 12
extent csum item
item 3 key (EXTENT_CSUM EXTENT_CSUM 4278931456) itemoff 3843 itemsize 40
extent csum item
item 4 key (EXTENT_CSUM EXTENT_CSUM 4278976512) itemoff 3819 itemsize 24
extent csum item
item 5 key (EXTENT_CSUM EXTENT_CSUM 4279001088) itemoff 3815 itemsize 4
extent csum item
item 6 key (EXTENT_CSUM EXTENT_CSUM 4279005184) itemoff 3787 itemsize 28
extent csum item
item 7 key (EXTENT_CSUM EXTENT_CSUM 4279033856) itemoff 3715 itemsize 72
extent csum item
item 8 key (EXTENT_CSUM EXTENT_CSUM 4279107584) itemoff 3619 itemsize 96
extent csum item
item 9 key (EXTENT_CSUM EXTENT_CSUM 72998785024) itemoff 3599 itemsize 20
extent csum item
item 10 key (EXTENT_CSUM EXTENT_CSUM 4279345152) itemoff 3583 itemsize 16
extent csum item
item 11 key (EXTENT_CSUM EXTENT_CSUM 4279369728) itemoff 3575 itemsize 8
extent csum item
item 12 key (EXTENT_CSUM EXTENT_CSUM 4279463936) itemoff 3551 itemsize 24
extent csum item
item 13 key (EXTENT_CSUM EXTENT_CSUM 4279496704) itemoff 3523 itemsize 28
extent csum item
item 14 key (EXTENT_CSUM EXTENT_CSUM 4279525376) itemoff 3515 itemsize 8
extent csum item
item 15 key (EXTENT_CSUM EXTENT_CSUM 4279533568) itemoff 3511 itemsize 4
extent csum item
item 16 key (EXTENT_CSUM EXTENT_CSUM 4279537664) itemoff 3491 itemsize 20
extent csum item
item 17 key (EXTENT_CSUM EXTENT_CSUM 4279562240) itemoff 3463 itemsize 28
extent csum item
item 18 key (EXTENT_CSUM EXTENT_CSUM 4279607296) itemoff 3459 itemsize 4
extent csum item
item 19 key (EXTENT_CSUM EXTENT_CSUM 4279615488) itemoff 3403 itemsize 56
extent csum item
item 20 key (EXTENT_CSUM EXTENT_CSUM 4280020992) itemoff 3395 itemsize 8
extent csum item
item 21 key (EXTENT_CSUM EXTENT_CSUM 4280033280) itemoff 3391 itemsize 4
extent csum item
item 22 key (EXTENT_CSUM EXTENT_CSUM 4280053760) itemoff 3379 itemsize 12
extent csum item
item 23 key (EXTENT_CSUM EXTENT_CSUM 4280082432) itemoff 3331 itemsize 48
extent csum item
item 24 key (EXTENT_CSUM EXTENT_CSUM 4280135680) itemoff 3311 itemsize 20
extent csum item
item 25 key (EXTENT_CSUM EXTENT_CSUM 4280156160) itemoff 3299 itemsize 12
extent csum item
item 26 key (EXTENT_CSUM EXTENT_CSUM 4280168448) itemoff 3255 itemsize 44
extent csum item
item 27 key (EXTENT_CSUM EXTENT_CSUM 4280229888) itemoff 3243 itemsize 12
extent csum item
item 28 key (EXTENT_CSUM EXTENT_CSUM 4280262656) itemoff 3231 itemsize 12
extent csum item
item 29 key (EXTENT_CSUM EXTENT_CSUM 4280360960) itemoff 3223 itemsize 8
extent csum item
item 30 key (EXTENT_CSUM EXTENT_CSUM 4280369152) itemoff 3123 itemsize 100
extent csum item
item 31 key (EXTENT_CSUM EXTENT_CSUM 4280496128) itemoff 3115 itemsize 8
extent csum item
item 32 key (EXTENT_CSUM EXTENT_CSUM 4280504320) itemoff 3103 itemsize 12
extent csum item
item 33 key (EXTENT_CSUM EXTENT_CSUM 4280532992) itemoff 3087 itemsize 16
extent csum item
item 34 key (EXTENT_CSUM EXTENT_CSUM 4280549376) itemoff 3071 itemsize 16
extent csum item
item 35 key (EXTENT_CSUM EXTENT_CSUM 4280565760) itemoff 3063 itemsize 8
extent csum item
item 36 key (EXTENT_CSUM EXTENT_CSUM 4280573952) itemoff 3055 itemsize 8
extent csum item
item 37 key (EXTENT_CSUM EXTENT_CSUM 4280664064) itemoff 3047 itemsize 8
extent csum item
item 38 key (EXTENT_CSUM EXTENT_CSUM 4280672256) itemoff 3023 itemsize 24
extent csum item
item 39 key (EXTENT_CSUM EXTENT_CSUM 4280696832) itemoff 2967 itemsize 56
extent csum item
item 40 key (EXTENT_CSUM EXTENT_CSUM 4280762368) itemoff 2947 itemsize 20
extent csum item
item 41 key (EXTENT_CSUM EXTENT_CSUM 4280852480) itemoff 2939 itemsize 8
extent csum item
item 42 key (EXTENT_CSUM EXTENT_CSUM 4280868864) itemoff 2931 itemsize 8
extent csum item
item 43 key (EXTENT_CSUM EXTENT_CSUM 4280885248) itemoff 2915 itemsize 16
extent csum item
item 44 key (EXTENT_CSUM EXTENT_CSUM 4280905728) itemoff 2911 itemsize 4
extent csum item
item 45 key (EXTENT_CSUM EXTENT_CSUM 4280909824) itemoff 2895 itemsize 16
extent csum item
item 46 key (EXTENT_CSUM EXTENT_CSUM 4280971264) itemoff 2859 itemsize 36
extent csum item
item 47 key (EXTENT_CSUM EXTENT_CSUM 4281036800) itemoff 2823 itemsize 36
extent csum item
item 48 key (EXTENT_CSUM EXTENT_CSUM 4281102336) itemoff 2771 itemsize 52
extent csum item
item 49 key (EXTENT_CSUM EXTENT_CSUM 4281176064) itemoff 2759 itemsize 12
extent csum item
item 50 key (EXTENT_CSUM EXTENT_CSUM 4281188352) itemoff 2743 itemsize 16
extent csum item
item 51 key (EXTENT_CSUM EXTENT_CSUM 4281233408) itemoff 2735 itemsize 8
extent csum item
item 52 key (EXTENT_CSUM EXTENT_CSUM 4281323520) itemoff 2723 itemsize 12
extent csum item
item 53 key (EXTENT_CSUM EXTENT_CSUM 4281352192) itemoff 2711 itemsize 12
extent csum item
item 54 key (EXTENT_CSUM EXTENT_CSUM 4281364480) itemoff 2707 itemsize 4
extent csum item
item 55 key (EXTENT_CSUM EXTENT_CSUM 4281376768) itemoff 2667 itemsize 40
extent csum item
item 56 key (EXTENT_CSUM EXTENT_CSUM 4281417728) itemoff 2611 itemsize 56
extent csum item
item 57 key (EXTENT_CSUM EXTENT_CSUM 4281475072) itemoff 2591 itemsize 20
extent csum item
item 58 key (EXTENT_CSUM EXTENT_CSUM 4281495552) itemoff 2555 itemsize 36
extent csum item
item 59 key (EXTENT_CSUM EXTENT_CSUM 4281532416) itemoff 2551 itemsize 4
extent csum item
item 60 key (EXTENT_CSUM EXTENT_CSUM 4281561088) itemoff 2547 itemsize 4
extent csum item
item 61 key (EXTENT_CSUM EXTENT_CSUM 4281565184) itemoff 2527 itemsize 20
extent csum item
item 62 key (EXTENT_CSUM EXTENT_CSUM 4281602048) itemoff 2511 itemsize 16
extent csum item
item 63 key (EXTENT_CSUM EXTENT_CSUM 4281638912) itemoff 2503 itemsize 8
extent csum item
item 64 key (EXTENT_CSUM EXTENT_CSUM 4281876480) itemoff 2455 itemsize 48
extent csum item
item 65 key (EXTENT_CSUM EXTENT_CSUM 4281937920) itemoff 2431 itemsize 24
extent csum item
item 66 key (EXTENT_CSUM EXTENT_CSUM 4282073088) itemoff 2411 itemsize 20
extent csum item
item 67 key (EXTENT_CSUM EXTENT_CSUM 4282093568) itemoff 2395 itemsize 16
extent csum item
item 68 key (EXTENT_CSUM EXTENT_CSUM 4282109952) itemoff 2387 itemsize 8
extent csum item
item 69 key (EXTENT_CSUM EXTENT_CSUM 4282134528) itemoff 2335 itemsize 52
extent csum item
item 70 key (EXTENT_CSUM EXTENT_CSUM 4282191872) itemoff 2331 itemsize 4
extent csum item
item 71 key (EXTENT_CSUM EXTENT_CSUM 4282216448) itemoff 2255 itemsize 76
extent csum item
item 72 key (EXTENT_CSUM EXTENT_CSUM 4282298368) itemoff 2251 itemsize 4
extent csum item
item 73 key (EXTENT_CSUM EXTENT_CSUM 4282503168) itemoff 2223 itemsize 28
extent csum item
item 74 key (EXTENT_CSUM EXTENT_CSUM 4282531840) itemoff 2211 itemsize 12
extent csum item
item 75 key (EXTENT_CSUM EXTENT_CSUM 4282548224) itemoff 2151 itemsize 60
extent csum item
item 76 key (EXTENT_CSUM EXTENT_CSUM 4282626048) itemoff 2123 itemsize 28
extent csum item
item 77 key (EXTENT_CSUM EXTENT_CSUM 4282658816) itemoff 2119 itemsize 4
extent csum item
item 78 key (EXTENT_CSUM EXTENT_CSUM 4282662912) itemoff 2107 itemsize 12
extent csum item
item 79 key (EXTENT_CSUM EXTENT_CSUM 4282679296) itemoff 2095 itemsize 12
extent csum item
item 80 key (EXTENT_CSUM EXTENT_CSUM 4282712064) itemoff 2071 itemsize 24
extent csum item
--
With respect,
Roman
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: btrfs crash with a corrupted(?) filesystem
2014-02-04 16:35 ` Roman Mamedov
@ 2014-02-04 16:40 ` Hugo Mills
0 siblings, 0 replies; 4+ messages in thread
From: Hugo Mills @ 2014-02-04 16:40 UTC (permalink / raw)
To: Roman Mamedov; +Cc: linux-btrfs
[-- Attachment #1: Type: text/plain, Size: 3072 bytes --]
On Tue, Feb 04, 2014 at 10:35:06PM +0600, Roman Mamedov wrote:
> On Tue, 4 Feb 2014 16:32:35 +0000
> Hugo Mills <hugo@carfax.org.uk> wrote:
>
> > On Tue, Feb 04, 2014 at 10:23:10PM +0600, Roman Mamedov wrote:
> > > Hello,
> > >
> > > My server had a period of instability (PSU-related issues), some lockups,
> > > some strange crashes, and some files became corrupted, and perhaps parts of
> > > a filesystem too. One BTRFS partition now fails with the following errors.
> > >
> > > On an attempt to make a snapshot:
> > >
> > > [ 48.035664] btrfs: corrupt leaf, bad key order: block=193529446400,root=1, slot=9
> > [snip]
> >
> > Bad key order is pretty much always down to hardware corrupting
> > data at some point -- which would go well with your list of hardware
> > problems above.
> >
> > > Currently I have it mounted read-only, and all data seems to be accessible.
> > > Short of copying everything away and recreating the FS, how can I bring it to
> > > a working order. Is btrfsck a good option here?
> >
> > The first investigation to do would be to look at the block in
> > question and see if it's got an obvious problem with it. If you post
> > the output of "btrfs-debug-tree -b 193529446400 /dev/whatever", we can
> > take a look at the indexing.
>
> Thanks; here it is:
>
> # btrfs-debug-tree -b 193529446400 /dev/md4
> leaf 193529446400 items 81 free space 46 generation 565572 owner 7
> fs uuid dd12de99-bbe5-45cf-b869-6313c1f58431
> chunk uuid b61f845a-ada5-4bcf-b995-7c5e1affa63d
> item 0 key (EXTENT_CSUM EXTENT_CSUM 4278808576) itemoff 3955 itemsize 40
> extent csum item
> item 1 key (EXTENT_CSUM EXTENT_CSUM 4278853632) itemoff 3895 itemsize 60
> extent csum item
> item 2 key (EXTENT_CSUM EXTENT_CSUM 4278919168) itemoff 3883 itemsize 12
> extent csum item
> item 3 key (EXTENT_CSUM EXTENT_CSUM 4278931456) itemoff 3843 itemsize 40
> extent csum item
> item 4 key (EXTENT_CSUM EXTENT_CSUM 4278976512) itemoff 3819 itemsize 24
> extent csum item
> item 5 key (EXTENT_CSUM EXTENT_CSUM 4279001088) itemoff 3815 itemsize 4
> extent csum item
> item 6 key (EXTENT_CSUM EXTENT_CSUM 4279005184) itemoff 3787 itemsize 28
> extent csum item
> item 7 key (EXTENT_CSUM EXTENT_CSUM 4279033856) itemoff 3715 itemsize 72
> extent csum item
> item 8 key (EXTENT_CSUM EXTENT_CSUM 4279107584) itemoff 3619 itemsize 96
> extent csum item
> item 9 key (EXTENT_CSUM EXTENT_CSUM 72998785024) itemoff 3599 itemsize 20
> extent csum item
^^^ Here it is.
The previous key (item 8):
>>> hex(4279107584)
'0xff0e0000L'
This key (item 9):
>>> hex(72998785024)
'0x10ff111000L'
So it looks likely that you've got a single bit flip in the key.
Josef had a patch for fsck some time before Christmas that would deal
with (some of) these cases, but I'm not sure if this is one of them.
Hugo.
--
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
PGP key: 65E74AC0 from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
--- You're never alone with a rubber duck... ---
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2014-02-04 16:40 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2014-02-04 16:23 btrfs crash with a corrupted(?) filesystem Roman Mamedov
2014-02-04 16:32 ` Hugo Mills
2014-02-04 16:35 ` Roman Mamedov
2014-02-04 16:40 ` Hugo Mills
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).