* Crash, boot mount failure: "corrupt leaf, slot offset bad"
@ 2016-01-04 17:05 Chris Bainbridge
2016-01-05 1:57 ` Qu Wenruo
0 siblings, 1 reply; 6+ messages in thread
From: Chris Bainbridge @ 2016-01-04 17:05 UTC (permalink / raw)
To: linux-btrfs
Kernel 4.4.0-rc7
System is Macbook with 109GB btrfs partition on SSD
System crashed (nothing in syslog, could have been btrfs or possibly GPU fault
shortly after running xrandr). After hard reset the btrfs partition was corrupt
and would not mount. I took an image of the partition (some of the output below
refers to sda5, some to loop0, it is the same image)
# dmesg of mount failure (using kernel 4.2 from an Ubuntu 15.10 recovery drive but same on 4.4.0-rc7):
[ 1969.425321] BTRFS critical (device sda5): corrupt leaf, slot offset bad: block=67239936,root=1, slot=82
[ 1969.428809] BTRFS critical (device sda5): corrupt leaf, slot offset bad: block=67239936,root=1, slot=82
[ 1969.431981] ------------[ cut here ]------------
[ 1969.432018] WARNING: CPU: 2 PID: 11162 at /build/linux-cRemOf/linux-4.2.0/fs/btrfs/extent-tree.c:6264 __btrfs_free_extent.isra.69+0x2ef/0xd70 [btrfs]()
[ 1969.432028] BTRFS: Transaction aborted (error -5)
[ 1969.432030] Modules linked in: drbg ansi_cprng ctr ccm intel_rapl iosf_mbi x86_pkg_temp_thermal intel_powerclamp btrfs arc4 xor coretemp rt2800usb rt2x00usb raid6_pq b43 rt2800lib rt2x00lib kvm_intel mac80211 cfg80211 btusb btrtl btbcm btintel bluetooth uvcvideo kvm ssb crc_ccitt crct10dif_pclmul crc32_pclmul snd_hda_codec_hdmi snd_hda_codec_cirrus snd_hda_codec_generic videobuf2_vmalloc videobuf2_memops videobuf2_core v4l2_common videodev snd_hda_intel snd_hda_codec snd_hda_core aesni_intel aes_x86_64 lrw gf128mul glue_helper applesmc snd_hwdep snd_pcm snd_timer snd soundcore joydev input_polldev media ablk_helper cryptd bcm5974 input_leds lpc_ich bcma mei_me mei thunderbolt sbs sbshc acpi_als kfifo_buf apple_gmux industrialio mac_hid shpchp apple_bl autofs4 hid_generic i915 hid_apple sdhci_pci
[ 1969.432115] i2c_algo_bit ahci uas drm_kms_helper libahci sdhci usb_storage usbhid drm video hid
[ 1969.432132] CPU: 2 PID: 11162 Comm: mount Tainted: G W 4.2.0-22-generic #27-Ubuntu
[ 1969.432136] Hardware name: Apple Inc. MacBookPro10,2/Mac-AFD8A9D944EA4843, BIOS MBP102.88Z.0106.B07.1501071215 01/07/2015
[ 1969.432139] 0000000000000000 00000000037e909d ffff88026267b5d8 ffffffff817e94c9
[ 1969.432145] 0000000000000000 ffff88026267b630 ffff88026267b618 ffffffff8107b3d6
[ 1969.432152] ffff88026267b608 0000000e5297c000 00000000fffffffb 0000000000000000
[ 1969.432157] Call Trace:
[ 1969.432167] [<ffffffff817e94c9>] dump_stack+0x45/0x57
[ 1969.432175] [<ffffffff8107b3d6>] warn_slowpath_common+0x86/0xc0
[ 1969.432181] [<ffffffff8107b465>] warn_slowpath_fmt+0x55/0x70
[ 1969.432199] [<ffffffffc086837f>] __btrfs_free_extent.isra.69+0x2ef/0xd70 [btrfs]
[ 1969.432229] [<ffffffffc08d1e5a>] ? find_ref_head+0x5a/0x80 [btrfs]
[ 1969.432248] [<ffffffffc086c718>] __btrfs_run_delayed_refs+0x988/0x1080 [btrfs]
[ 1969.432268] [<ffffffffc086fb1e>] btrfs_run_delayed_refs.part.73+0x6e/0x270 [btrfs]
[ 1969.432284] [<ffffffffc0859ec3>] ? btrfs_set_path_blocking+0x43/0x80 [btrfs]
[ 1969.432306] [<ffffffffc086fd35>] btrfs_run_delayed_refs+0x15/0x20 [btrfs]
[ 1969.432326] [<ffffffffc0884756>] btrfs_commit_transaction+0x56/0xb20 [btrfs]
[ 1969.432332] [<ffffffff811dde9f>] ? kmem_cache_free+0x1cf/0x1e0
[ 1969.432356] [<ffffffffc08c9e0d>] btrfs_recover_log_trees+0x3ed/0x490 [btrfs]
[ 1969.432378] [<ffffffffc08c5890>] ? replay_one_extent+0x6a0/0x6a0 [btrfs]
[ 1969.432397] [<ffffffffc0881b21>] open_ctree+0x19b1/0x23e0 [btrfs]
[ 1969.432412] [<ffffffffc0856fae>] btrfs_mount+0x94e/0xa70 [btrfs]
[ 1969.432420] [<ffffffff813da845>] ? find_next_bit+0x15/0x20
[ 1969.432427] [<ffffffff811a5e15>] ? pcpu_alloc+0x385/0x670
[ 1969.432434] [<ffffffff812011c8>] mount_fs+0x38/0x160
[ 1969.432439] [<ffffffff811a6135>] ? __alloc_percpu+0x15/0x20
[ 1969.432446] [<ffffffff8121cfab>] vfs_kern_mount+0x6b/0x120
[ 1969.432463] [<ffffffffc0856848>] btrfs_mount+0x1e8/0xa70 [btrfs]
[ 1969.432469] [<ffffffff811a5e15>] ? pcpu_alloc+0x385/0x670
[ 1969.432475] [<ffffffff812011c8>] mount_fs+0x38/0x160
[ 1969.432481] [<ffffffff811a6135>] ? __alloc_percpu+0x15/0x20
[ 1969.432486] [<ffffffff8121cfab>] vfs_kern_mount+0x6b/0x120
[ 1969.432493] [<ffffffff8121f7b6>] do_mount+0x246/0xd10
[ 1969.432498] [<ffffffff811a08fe>] ? strndup_user+0x4e/0xb0
[ 1969.432503] [<ffffffff811a0876>] ? memdup_user+0x46/0x80
[ 1969.432510] [<ffffffff812205af>] SyS_mount+0x9f/0x100
[ 1969.432519] [<ffffffff817f02b2>] entry_SYSCALL_64_fastpath+0x16/0x75
[ 1969.432523] ---[ end trace 7a560cc73341e0d1 ]---
[ 1969.432528] BTRFS: error (device sda5) in __btrfs_free_extent:6264: errno=-5 IO failure
[ 1969.436576] BTRFS: error (device sda5) in btrfs_run_delayed_refs:2781: errno=-5 IO failure
[ 1969.442036] BTRFS: error (device sda5) in btrfs_replay_log:2375: errno=-5 IO failure (Failed to recover log tree)
[ 1969.446087] BTRFS error (device sda5): cleaner transaction attach returned -30
[ 1969.486477] BTRFS: open_ctree failed
# btrfsck:
checking filesystem on sda5
UUID: b1103526-98a3-4b40-a782-cf66721ed600
checking extents
incorrect offsets 11897 5713478
bad block 67239936
Errors found in extent allocation tree or chunk allocation
checking free space cache
There is no free space entry for 61515563008-62297997312
cache appears valid but isnt 61224255488
found 7839154285 bytes used err is -22
total csum bytes: 0
total tree bytes: 7454720
total fs tree bytes: 0
total extent tree bytes: 7356416
btree space waste bytes: 2454021
file data blocks allocated: 28508160
referenced 28508160
# btrfs-image (compiled v4.3.1 from git:.../kdave/btrfs-progs) segfaults:
Program received signal SIGSEGV, Segmentation fault.
copy_from_extent_tree (path=0x64e420, metadump=0x7fffffffdf90) at btrfs-image.c:1265
1265 if (btrfs_extent_flags(leaf, ei) &
(gdb)
(gdb) bt
#0 copy_from_extent_tree (path=0x64e420, metadump=0x7fffffffdf90) at btrfs-image.c:1265
#1 create_metadump (walk_trees=<optimised out>, sanitize=<optimised out>, compress_level=<optimised out>, num_threads=0, out=0x64e010, input=0x7fffffffe3fe "/dev/sda5") at btrfs-image.c:1364
#2 main (argc=2, argv=0x7fffffffe198) at btrfs-image.c:2804
# After btrfs-zero-log the partition mounts but with errors:
[47995.081106] BTRFS: checking UUID tree
[48030.691448] BTRFS critical (device loop0): corrupt leaf, slot offset bad: block=67239936,root=1, slot=82
[48030.794656] BTRFS critical (device loop0): corrupt leaf, slot offset bad: block=67239936,root=1, slot=82
[49304.640667] BTRFS critical (device loop0): corrupt leaf, slot offset bad: block=67239936,root=1, slot=82
[49304.641861] ------------[ cut here ]------------
[49304.641888] WARNING: CPU: 0 PID: 3829 at /build/linux-cRemOf/linux-4.2.0/fs/btrfs/extent-tree.c:6264 __btrfs_free_extent.isra.69+0x2ef/0xd70 [btrfs]()
[49304.641890] BTRFS: Transaction aborted (error -5)
[49304.641892] Modules linked in: btrfs xor raid6_pq arc4 rtl8723be btcoexist intel_rapl rtl8723_common rtl_pci intel_soc_dts_iosf rtlwifi mac80211 intel_powerclamp cfg80211 snd_intel_sst_acpi snd_soc_rt5640 snd_intel_sst_core joydev input_leds snd_hda_codec_hdmi snd_soc_rl6231 snd_hda_codec_conexant snd_soc_sst_mfld_platform snd_hda_codec_generic rtsx_usb_ms coretemp kvm punit_atom_debug snd_soc_core memstick serio_raw crct10dif_pclmul crc32_pclmul cryptd snd_compress dw_dmac snd_hda_intel lpc_ich dw_dmac_core btusb btrtl btbcm btintel bluetooth ac97_bus iosf_mbi uvcvideo videobuf2_vmalloc videobuf2_memops videobuf2_core v4l2_common videodev media snd_hda_codec mei_txe shpchp snd_hda_core mei snd_hwdep snd_pcm_dmaengine rfkill_gpio 8250_fintek ideapad_laptop intel_smartconnect sparse_keymap snd_pcm
[49304.641945] snd_timer snd soundcore snd_soc_sst_acpi i2c_designware_platform pwm_lpss_platform 8250_dw i2c_designware_core spi_pxa2xx_platform pwm_lpss mac_hid autofs4 rtsx_usb_sdmmc rtsx_usb uas usb_storage psmouse i915 i2c_algo_bit r8169 drm_kms_helper mii drm ahci libahci video i2c_hid hid sdhci_acpi sdhci
[49304.641971] CPU: 0 PID: 3829 Comm: btrfs-transacti Tainted: G W 4.2.0-22-generic #27-Ubuntu
[49304.641973] Hardware name: LENOVO 80G0/Lancer 5A6, BIOS A7CN43WW 10/21/2014
[49304.641976] 0000000000000000 00000000dfc9f16b ffff880122ddfa98 ffffffff817e94c9
[49304.641980] 0000000000000000 ffff880122ddfaf0 ffff880122ddfad8 ffffffff8107b3d6
[49304.641983] ffff880122ddfac8 0000000e52860000 00000000fffffffb 0000000000000000
[49304.641987] Call Trace:
[49304.641995] [<ffffffff817e94c9>] dump_stack+0x45/0x57
[49304.642000] [<ffffffff8107b3d6>] warn_slowpath_common+0x86/0xc0
[49304.642004] [<ffffffff8107b465>] warn_slowpath_fmt+0x55/0x70
[49304.642017] [<ffffffffc07f737f>] __btrfs_free_extent.isra.69+0x2ef/0xd70 [btrfs]
[49304.642035] [<ffffffffc0860e5a>] ? find_ref_head+0x5a/0x80 [btrfs]
[49304.642049] [<ffffffffc07fb718>] __btrfs_run_delayed_refs+0x988/0x1080 [btrfs]
[49304.642063] [<ffffffffc07feb1e>] btrfs_run_delayed_refs.part.73+0x6e/0x270 [btrfs]
[49304.642081] [<ffffffffc08750a3>] ? btrfs_scrub_pause+0xe3/0x110 [btrfs]
[49304.642094] [<ffffffffc07fed35>] btrfs_run_delayed_refs+0x15/0x20 [btrfs]
[49304.642109] [<ffffffffc0813c79>] btrfs_commit_transaction+0x579/0xb20 [btrfs]
[49304.642124] [<ffffffffc080ee19>] transaction_kthread+0x229/0x240 [btrfs]
[49304.642139] [<ffffffffc080ebf0>] ? btrfs_cleanup_transaction+0x550/0x550 [btrfs]
[49304.642143] [<ffffffff8109a868>] kthread+0xd8/0xf0
[49304.642147] [<ffffffff8109a790>] ? kthread_create_on_node+0x1f0/0x1f0
[49304.642151] [<ffffffff817f06df>] ret_from_fork+0x3f/0x70
[49304.642154] [<ffffffff8109a790>] ? kthread_create_on_node+0x1f0/0x1f0
[49304.642156] ---[ end trace bba14eeb59d201a5 ]---
[49304.642160] BTRFS: error (device loop0) in __btrfs_free_extent:6264: errno=-5 IO failure
[49304.643386] BTRFS info (device loop0): forced readonly
[49304.643391] BTRFS: error (device loop0) in btrfs_run_delayed_refs:2781: errno=-5 IO failure
[49304.644628] BTRFS warning (device loop0): Skipping commit of aborted transaction.
[49304.644632] BTRFS: error (device loop0) in cleanup_transaction:1710: errno=-5 IO failure
# btrfs fi show /mnt/x:
Label: 'LINUX' uuid: b1103526-98a3-4b40-a782-cf66721ed600
Total devices 1 FS bytes used 84.16GiB
devid 1 size 108.80GiB used 108.80GiB path /dev/loop0
# btrfs fi df /mnt/x:
Data, single: total=106.79GiB, used=83.00GiB
System, single: total=4.00MiB, used=16.00KiB
Metadata, single: total=2.01GiB, used=1.16GiB
GlobalReserve, single: total=400.00MiB, used=0.00B
# btrfsck --repair:
enabling repair mode
Checking filesystem on /dev/sda5
UUID: b1103526-98a3-4b40-a782-cf66721ed600
checking extents
incorrect offsets 11897 5713478
Deleting bogus item [61515546624,168,16384] at slot 83 on block 4243456
Shifting item nr 83 by 275 bytes in block 4243456
items overlap, can't fix
cmds-check.c:4118: fix_item_offset: Assertion `ret` failed.
btrfs check[0x426e9c]
btrfs check[0x427114]
btrfs check[0x427d2d]
btrfs check(cmd_check+0x15bb)[0x42b203]
btrfs check(main+0x155)[0x409eae]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0)[0x7ff6aacb2a40]
btrfs check(_start+0x29)[0x409a69]
# btrfsck --repair --init-extent-tree:
[takes a while, over 3 million lines of output... small extract of end follows]
Backref 123269292032 root 257 owner 1089423 offset 17063936 num_refs 0 not found in extent tree
Incorrect local backref count on 123269292032 root 257 owner 1089423 offset 17063936 found 1 wanted 0 back 0x25c5ae0
Backref 123269292032 parent 61250600960 owner 0 offset 0 num_refs 0 not found in extent tree
Incorrect local backref count on 123269292032 parent 61250600960 owner 0 offset 0 found 1 wanted 0 back 0xdadeb40
backpointer mismatch on [123269292032 4096]
ref mismatch on [123269373952 12288] extent item 0, found 3
adding new data backref on 123269373952 root 257 owner 184332 offset 356352 found 1
adding new data backref on 123269373952 parent 61254959104 owner 0 offset 0 found 1
adding new data backref on 123269373952 parent 63946752 owner 0 offset 0 found 1
Backref 123269373952 root 257 owner 184332 offset 356352 num_refs 0 not found in extent tree
Incorrect local backref count on 123269373952 root 257 owner 184332 offset 356352 found 1 wanted 0 back 0x258a4e0
Backref 123269373952 parent 61254959104 owner 0 offset 0 num_refs 0 not found in extent tree
Incorrect local backref count on 123269373952 parent 61254959104 owner 0 offset 0 found 1 wanted 0 back 0xdb085a0
Backref 123269373952 parent 63946752 owner 0 offset 0 num_refs 0 not found in extent tree
Incorrect local backref count on 123269373952 parent 63946752 owner 0 offset 0 found 1 wanted 0 back 0xde73e50
backpointer mismatch on [123269373952 12288]
ref mismatch on [123269513216 28672] extent item 0, found 1
adding new data backref on 123269513216 root 260 owner 407786 offset 559493120 found 1
Backref 123269513216 root 260 owner 407786 offset 559493120 num_refs 0 not found in extent tree
Incorrect local backref count on 123269513216 root 260 owner 407786 offset 559493120 found 1 wanted 0 back 0xca364a0
backpointer mismatch on [123269513216 28672]
repaired damaged extent references
Fixed 0 roots.
checking free space cache
cache and super generation don't match, space cache will be invalidated
checking fs roots
checking csums
checking root refs
found 1168523045 bytes used err is 0
total csum bytes: 84470212
total tree bytes: 1167949824
total fs tree bytes: 1050476544
total extent tree bytes: 16384
btree space waste bytes: 231805196
file data blocks allocated: 209697452032
referenced 92010127360
# btrfs scrub then showed only a few files with bad checksums:
- 5.8GB @jessie/var/log/syslog.1
- 308MB @jessie/var/log/syslog.1.gz
- 815MB @tmp/linux/.git/objects/pack/tmp_pack_M544ZX
The timestamps show these files are several months old so were probably not
open at the time of the crash.
After removing these files no other errors are reported.
The recovered partition then successfully booted, but the next boot failed with
drive full errors. It looks like a btrfs metadata issue:
Data, single: total=106.79GiB, used=82.01GiB
System, single: total=4.00MiB, used=16.00KiB
Metadata, single: total=2.01GiB, used=1.51GiB
GlobalReserve, single: total=512.00MiB, used=0.00B
Even though the original pre-recovery crashed paritition showed 1.16GiB/2.01GiB
metadata used, it appears used metadata can quickly grow.
Deleting some subvolumes brought the metadata used down to 1.02GiB but:
# btrfs balance start -dlimit=5 results in "ERROR: error during balancing '/' - No space left on device"
Suggests that even with a used/total metadata of 1.02GiB/2.01GiB there are
still issues. Perhaps the used portion can quickly grow to the point where it
causes problems?
Repeating balance again, starting with dlimit=1 then dlimit=2 then dlimit=5,
was successful with 8 chunks reallocated and "btrfs fi show /" showing
103.02GiB/108.80GiB allocated.
I have an image of the crashed partition so let me know if any further analysis
would be useful. I would guess that the low free metadata may have been a
factor in the crash; it was enough of an issue to cause the post-recovery
system to be unable to boot properly.
FWIW I found several other reports of unbootable btrfs with the same "corrupt
leaf, slot offset bad" error, some also reporting that it occurred after a
crash that required a hard reset:
https://bugzilla.kernel.org/show_bug.cgi?id=74611
http://unix.stackexchange.com/questions/195888/how-to-repair-a-btrfs-block
http://www.spinics.net/lists/linux-btrfs/msg48167.html
http://ubuntuforums.org/showthread.php?t=2301288
https://lists.archlinux.org/pipermail/arch-general/2014-April/035885.html
http://comments.gmane.org/gmane.comp.file-systems.btrfs/34190
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Crash, boot mount failure: "corrupt leaf, slot offset bad"
2016-01-04 17:05 Crash, boot mount failure: "corrupt leaf, slot offset bad" Chris Bainbridge
@ 2016-01-05 1:57 ` Qu Wenruo
2016-01-05 13:41 ` Chris Bainbridge
0 siblings, 1 reply; 6+ messages in thread
From: Qu Wenruo @ 2016-01-05 1:57 UTC (permalink / raw)
To: Chris Bainbridge, linux-btrfs
Chris Bainbridge wrote on 2016/01/04 17:05 +0000:
> Kernel 4.4.0-rc7
> System is Macbook with 109GB btrfs partition on SSD
>
> System crashed (nothing in syslog, could have been btrfs or possibly GPU fault
> shortly after running xrandr). After hard reset the btrfs partition was corrupt
> and would not mount. I took an image of the partition (some of the output below
> refers to sda5, some to loop0, it is the same image)
>
> # dmesg of mount failure (using kernel 4.2 from an Ubuntu 15.10 recovery drive but same on 4.4.0-rc7):
>
> [ 1969.425321] BTRFS critical (device sda5): corrupt leaf, slot offset bad: block=67239936,root=1, slot=82
> [ 1969.428809] BTRFS critical (device sda5): corrupt leaf, slot offset bad: block=67239936,root=1, slot=82
> [ 1969.431981] ------------[ cut here ]------------
> [ 1969.432018] WARNING: CPU: 2 PID: 11162 at /build/linux-cRemOf/linux-4.2.0/fs/btrfs/extent-tree.c:6264 __btrfs_free_extent.isra.69+0x2ef/0xd70 [btrfs]()
> [ 1969.432028] BTRFS: Transaction aborted (error -5)
> [ 1969.432030] Modules linked in: drbg ansi_cprng ctr ccm intel_rapl iosf_mbi x86_pkg_temp_thermal intel_powerclamp btrfs arc4 xor coretemp rt2800usb rt2x00usb raid6_pq b43 rt2800lib rt2x00lib kvm_intel mac80211 cfg80211 btusb btrtl btbcm btintel bluetooth uvcvideo kvm ssb crc_ccitt crct10dif_pclmul crc32_pclmul snd_hda_codec_hdmi snd_hda_codec_cirrus snd_hda_codec_generic videobuf2_vmalloc videobuf2_memops videobuf2_core v4l2_common videodev snd_hda_intel snd_hda_codec snd_hda_core aesni_intel aes_x86_64 lrw gf128mul glue_helper applesmc snd_hwdep snd_pcm snd_timer snd soundcore joydev input_polldev media ablk_helper cryptd bcm5974 input_leds lpc_ich bcma mei_me mei thunderbolt sbs sbshc acpi_als kfifo_buf apple_gmux industrialio mac_hid shpchp apple_bl autofs4 hid_generic i915 hid_apple sdhci_pci
> [ 1969.432115] i2c_algo_bit ahci uas drm_kms_helper libahci sdhci usb_storage usbhid drm video hid
> [ 1969.432132] CPU: 2 PID: 11162 Comm: mount Tainted: G W 4.2.0-22-generic #27-Ubuntu
> [ 1969.432136] Hardware name: Apple Inc. MacBookPro10,2/Mac-AFD8A9D944EA4843, BIOS MBP102.88Z.0106.B07.1501071215 01/07/2015
> [ 1969.432139] 0000000000000000 00000000037e909d ffff88026267b5d8 ffffffff817e94c9
> [ 1969.432145] 0000000000000000 ffff88026267b630 ffff88026267b618 ffffffff8107b3d6
> [ 1969.432152] ffff88026267b608 0000000e5297c000 00000000fffffffb 0000000000000000
> [ 1969.432157] Call Trace:
> [ 1969.432167] [<ffffffff817e94c9>] dump_stack+0x45/0x57
> [ 1969.432175] [<ffffffff8107b3d6>] warn_slowpath_common+0x86/0xc0
> [ 1969.432181] [<ffffffff8107b465>] warn_slowpath_fmt+0x55/0x70
> [ 1969.432199] [<ffffffffc086837f>] __btrfs_free_extent.isra.69+0x2ef/0xd70 [btrfs]
> [ 1969.432229] [<ffffffffc08d1e5a>] ? find_ref_head+0x5a/0x80 [btrfs]
> [ 1969.432248] [<ffffffffc086c718>] __btrfs_run_delayed_refs+0x988/0x1080 [btrfs]
> [ 1969.432268] [<ffffffffc086fb1e>] btrfs_run_delayed_refs.part.73+0x6e/0x270 [btrfs]
> [ 1969.432284] [<ffffffffc0859ec3>] ? btrfs_set_path_blocking+0x43/0x80 [btrfs]
> [ 1969.432306] [<ffffffffc086fd35>] btrfs_run_delayed_refs+0x15/0x20 [btrfs]
> [ 1969.432326] [<ffffffffc0884756>] btrfs_commit_transaction+0x56/0xb20 [btrfs]
> [ 1969.432332] [<ffffffff811dde9f>] ? kmem_cache_free+0x1cf/0x1e0
> [ 1969.432356] [<ffffffffc08c9e0d>] btrfs_recover_log_trees+0x3ed/0x490 [btrfs]
> [ 1969.432378] [<ffffffffc08c5890>] ? replay_one_extent+0x6a0/0x6a0 [btrfs]
> [ 1969.432397] [<ffffffffc0881b21>] open_ctree+0x19b1/0x23e0 [btrfs]
> [ 1969.432412] [<ffffffffc0856fae>] btrfs_mount+0x94e/0xa70 [btrfs]
> [ 1969.432420] [<ffffffff813da845>] ? find_next_bit+0x15/0x20
> [ 1969.432427] [<ffffffff811a5e15>] ? pcpu_alloc+0x385/0x670
> [ 1969.432434] [<ffffffff812011c8>] mount_fs+0x38/0x160
> [ 1969.432439] [<ffffffff811a6135>] ? __alloc_percpu+0x15/0x20
> [ 1969.432446] [<ffffffff8121cfab>] vfs_kern_mount+0x6b/0x120
> [ 1969.432463] [<ffffffffc0856848>] btrfs_mount+0x1e8/0xa70 [btrfs]
> [ 1969.432469] [<ffffffff811a5e15>] ? pcpu_alloc+0x385/0x670
> [ 1969.432475] [<ffffffff812011c8>] mount_fs+0x38/0x160
> [ 1969.432481] [<ffffffff811a6135>] ? __alloc_percpu+0x15/0x20
> [ 1969.432486] [<ffffffff8121cfab>] vfs_kern_mount+0x6b/0x120
> [ 1969.432493] [<ffffffff8121f7b6>] do_mount+0x246/0xd10
> [ 1969.432498] [<ffffffff811a08fe>] ? strndup_user+0x4e/0xb0
> [ 1969.432503] [<ffffffff811a0876>] ? memdup_user+0x46/0x80
> [ 1969.432510] [<ffffffff812205af>] SyS_mount+0x9f/0x100
> [ 1969.432519] [<ffffffff817f02b2>] entry_SYSCALL_64_fastpath+0x16/0x75
> [ 1969.432523] ---[ end trace 7a560cc73341e0d1 ]---
> [ 1969.432528] BTRFS: error (device sda5) in __btrfs_free_extent:6264: errno=-5 IO failure
> [ 1969.436576] BTRFS: error (device sda5) in btrfs_run_delayed_refs:2781: errno=-5 IO failure
> [ 1969.442036] BTRFS: error (device sda5) in btrfs_replay_log:2375: errno=-5 IO failure (Failed to recover log tree)
> [ 1969.446087] BTRFS error (device sda5): cleaner transaction attach returned -30
> [ 1969.486477] BTRFS: open_ctree failed
>
>
> # btrfsck:
>
> checking filesystem on sda5
> UUID: b1103526-98a3-4b40-a782-cf66721ed600
> checking extents
> incorrect offsets 11897 5713478
> bad block 67239936
> Errors found in extent allocation tree or chunk allocation
> checking free space cache
> There is no free space entry for 61515563008-62297997312
> cache appears valid but isnt 61224255488
> found 7839154285 bytes used err is -22
> total csum bytes: 0
> total tree bytes: 7454720
> total fs tree bytes: 0
> total extent tree bytes: 7356416
> btree space waste bytes: 2454021
> file data blocks allocated: 28508160
> referenced 28508160
>
>
> # btrfs-image (compiled v4.3.1 from git:.../kdave/btrfs-progs) segfaults:
>
> Program received signal SIGSEGV, Segmentation fault.
> copy_from_extent_tree (path=0x64e420, metadump=0x7fffffffdf90) at btrfs-image.c:1265
> 1265 if (btrfs_extent_flags(leaf, ei) &
> (gdb)
> (gdb) bt
> #0 copy_from_extent_tree (path=0x64e420, metadump=0x7fffffffdf90) at btrfs-image.c:1265
> #1 create_metadump (walk_trees=<optimised out>, sanitize=<optimised out>, compress_level=<optimised out>, num_threads=0, out=0x64e010, input=0x7fffffffe3fe "/dev/sda5") at btrfs-image.c:1364
> #2 main (argc=2, argv=0x7fffffffe198) at btrfs-image.c:2804
>
>
> # After btrfs-zero-log the partition mounts but with errors:
>
> [47995.081106] BTRFS: checking UUID tree
> [48030.691448] BTRFS critical (device loop0): corrupt leaf, slot offset bad: block=67239936,root=1, slot=82
> [48030.794656] BTRFS critical (device loop0): corrupt leaf, slot offset bad: block=67239936,root=1, slot=82
> [49304.640667] BTRFS critical (device loop0): corrupt leaf, slot offset bad: block=67239936,root=1, slot=82
> [49304.641861] ------------[ cut here ]------------
> [49304.641888] WARNING: CPU: 0 PID: 3829 at /build/linux-cRemOf/linux-4.2.0/fs/btrfs/extent-tree.c:6264 __btrfs_free_extent.isra.69+0x2ef/0xd70 [btrfs]()
> [49304.641890] BTRFS: Transaction aborted (error -5)
> [49304.641892] Modules linked in: btrfs xor raid6_pq arc4 rtl8723be btcoexist intel_rapl rtl8723_common rtl_pci intel_soc_dts_iosf rtlwifi mac80211 intel_powerclamp cfg80211 snd_intel_sst_acpi snd_soc_rt5640 snd_intel_sst_core joydev input_leds snd_hda_codec_hdmi snd_soc_rl6231 snd_hda_codec_conexant snd_soc_sst_mfld_platform snd_hda_codec_generic rtsx_usb_ms coretemp kvm punit_atom_debug snd_soc_core memstick serio_raw crct10dif_pclmul crc32_pclmul cryptd snd_compress dw_dmac snd_hda_intel lpc_ich dw_dmac_core btusb btrtl btbcm btintel bluetooth ac97_bus iosf_mbi uvcvideo videobuf2_vmalloc videobuf2_memops videobuf2_core v4l2_common videodev media snd_hda_codec mei_txe shpchp snd_hda_core mei snd_hwdep snd_pcm_dmaengine rfkill_gpio 8250_fintek ideapad_laptop intel_smartconnect sparse_keymap snd_pcm
> [49304.641945] snd_timer snd soundcore snd_soc_sst_acpi i2c_designware_platform pwm_lpss_platform 8250_dw i2c_designware_core spi_pxa2xx_platform pwm_lpss mac_hid autofs4 rtsx_usb_sdmmc rtsx_usb uas usb_storage psmouse i915 i2c_algo_bit r8169 drm_kms_helper mii drm ahci libahci video i2c_hid hid sdhci_acpi sdhci
> [49304.641971] CPU: 0 PID: 3829 Comm: btrfs-transacti Tainted: G W 4.2.0-22-generic #27-Ubuntu
> [49304.641973] Hardware name: LENOVO 80G0/Lancer 5A6, BIOS A7CN43WW 10/21/2014
> [49304.641976] 0000000000000000 00000000dfc9f16b ffff880122ddfa98 ffffffff817e94c9
> [49304.641980] 0000000000000000 ffff880122ddfaf0 ffff880122ddfad8 ffffffff8107b3d6
> [49304.641983] ffff880122ddfac8 0000000e52860000 00000000fffffffb 0000000000000000
> [49304.641987] Call Trace:
> [49304.641995] [<ffffffff817e94c9>] dump_stack+0x45/0x57
> [49304.642000] [<ffffffff8107b3d6>] warn_slowpath_common+0x86/0xc0
> [49304.642004] [<ffffffff8107b465>] warn_slowpath_fmt+0x55/0x70
> [49304.642017] [<ffffffffc07f737f>] __btrfs_free_extent.isra.69+0x2ef/0xd70 [btrfs]
> [49304.642035] [<ffffffffc0860e5a>] ? find_ref_head+0x5a/0x80 [btrfs]
> [49304.642049] [<ffffffffc07fb718>] __btrfs_run_delayed_refs+0x988/0x1080 [btrfs]
> [49304.642063] [<ffffffffc07feb1e>] btrfs_run_delayed_refs.part.73+0x6e/0x270 [btrfs]
> [49304.642081] [<ffffffffc08750a3>] ? btrfs_scrub_pause+0xe3/0x110 [btrfs]
> [49304.642094] [<ffffffffc07fed35>] btrfs_run_delayed_refs+0x15/0x20 [btrfs]
> [49304.642109] [<ffffffffc0813c79>] btrfs_commit_transaction+0x579/0xb20 [btrfs]
> [49304.642124] [<ffffffffc080ee19>] transaction_kthread+0x229/0x240 [btrfs]
> [49304.642139] [<ffffffffc080ebf0>] ? btrfs_cleanup_transaction+0x550/0x550 [btrfs]
> [49304.642143] [<ffffffff8109a868>] kthread+0xd8/0xf0
> [49304.642147] [<ffffffff8109a790>] ? kthread_create_on_node+0x1f0/0x1f0
> [49304.642151] [<ffffffff817f06df>] ret_from_fork+0x3f/0x70
> [49304.642154] [<ffffffff8109a790>] ? kthread_create_on_node+0x1f0/0x1f0
> [49304.642156] ---[ end trace bba14eeb59d201a5 ]---
> [49304.642160] BTRFS: error (device loop0) in __btrfs_free_extent:6264: errno=-5 IO failure
> [49304.643386] BTRFS info (device loop0): forced readonly
> [49304.643391] BTRFS: error (device loop0) in btrfs_run_delayed_refs:2781: errno=-5 IO failure
> [49304.644628] BTRFS warning (device loop0): Skipping commit of aborted transaction.
> [49304.644632] BTRFS: error (device loop0) in cleanup_transaction:1710: errno=-5 IO failure
>
>
> # btrfs fi show /mnt/x:
>
> Label: 'LINUX' uuid: b1103526-98a3-4b40-a782-cf66721ed600
> Total devices 1 FS bytes used 84.16GiB
> devid 1 size 108.80GiB used 108.80GiB path /dev/loop0
>
>
> # btrfs fi df /mnt/x:
>
> Data, single: total=106.79GiB, used=83.00GiB
> System, single: total=4.00MiB, used=16.00KiB
> Metadata, single: total=2.01GiB, used=1.16GiB
> GlobalReserve, single: total=400.00MiB, used=0.00B
>
>
> # btrfsck --repair:
>
> enabling repair mode
> Checking filesystem on /dev/sda5
> UUID: b1103526-98a3-4b40-a782-cf66721ed600
> checking extents
> incorrect offsets 11897 5713478
> Deleting bogus item [61515546624,168,16384] at slot 83 on block 4243456
> Shifting item nr 83 by 275 bytes in block 4243456
> items overlap, can't fix
> cmds-check.c:4118: fix_item_offset: Assertion `ret` failed.
> btrfs check[0x426e9c]
> btrfs check[0x427114]
> btrfs check[0x427d2d]
> btrfs check(cmd_check+0x15bb)[0x42b203]
> btrfs check(main+0x155)[0x409eae]
> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0)[0x7ff6aacb2a40]
> btrfs check(_start+0x29)[0x409a69]
>
>
> # btrfsck --repair --init-extent-tree:
>
> [takes a while, over 3 million lines of output... small extract of end follows]
>
> Backref 123269292032 root 257 owner 1089423 offset 17063936 num_refs 0 not found in extent tree
> Incorrect local backref count on 123269292032 root 257 owner 1089423 offset 17063936 found 1 wanted 0 back 0x25c5ae0
> Backref 123269292032 parent 61250600960 owner 0 offset 0 num_refs 0 not found in extent tree
> Incorrect local backref count on 123269292032 parent 61250600960 owner 0 offset 0 found 1 wanted 0 back 0xdadeb40
> backpointer mismatch on [123269292032 4096]
> ref mismatch on [123269373952 12288] extent item 0, found 3
> adding new data backref on 123269373952 root 257 owner 184332 offset 356352 found 1
> adding new data backref on 123269373952 parent 61254959104 owner 0 offset 0 found 1
> adding new data backref on 123269373952 parent 63946752 owner 0 offset 0 found 1
> Backref 123269373952 root 257 owner 184332 offset 356352 num_refs 0 not found in extent tree
> Incorrect local backref count on 123269373952 root 257 owner 184332 offset 356352 found 1 wanted 0 back 0x258a4e0
> Backref 123269373952 parent 61254959104 owner 0 offset 0 num_refs 0 not found in extent tree
> Incorrect local backref count on 123269373952 parent 61254959104 owner 0 offset 0 found 1 wanted 0 back 0xdb085a0
> Backref 123269373952 parent 63946752 owner 0 offset 0 num_refs 0 not found in extent tree
> Incorrect local backref count on 123269373952 parent 63946752 owner 0 offset 0 found 1 wanted 0 back 0xde73e50
> backpointer mismatch on [123269373952 12288]
> ref mismatch on [123269513216 28672] extent item 0, found 1
> adding new data backref on 123269513216 root 260 owner 407786 offset 559493120 found 1
> Backref 123269513216 root 260 owner 407786 offset 559493120 num_refs 0 not found in extent tree
> Incorrect local backref count on 123269513216 root 260 owner 407786 offset 559493120 found 1 wanted 0 back 0xca364a0
> backpointer mismatch on [123269513216 28672]
> repaired damaged extent references
> Fixed 0 roots.
> checking free space cache
> cache and super generation don't match, space cache will be invalidated
> checking fs roots
> checking csums
> checking root refs
> found 1168523045 bytes used err is 0
> total csum bytes: 84470212
> total tree bytes: 1167949824
> total fs tree bytes: 1050476544
> total extent tree bytes: 16384
> btree space waste bytes: 231805196
> file data blocks allocated: 209697452032
> referenced 92010127360
>
Glad that btrfsck --repair worked for you.
>
> # btrfs scrub then showed only a few files with bad checksums:
>
> - 5.8GB @jessie/var/log/syslog.1
> - 308MB @jessie/var/log/syslog.1.gz
> - 815MB @tmp/linux/.git/objects/pack/tmp_pack_M544ZX
>
> The timestamps show these files are several months old so were probably not
> open at the time of the crash.
>
> After removing these files no other errors are reported.
>
> The recovered partition then successfully booted, but the next boot failed with
> drive full errors. It looks like a btrfs metadata issue:
>
> Data, single: total=106.79GiB, used=82.01GiB
> System, single: total=4.00MiB, used=16.00KiB
> Metadata, single: total=2.01GiB, used=1.51GiB
> GlobalReserve, single: total=512.00MiB, used=0.00B
That's the btrfs fi df misleading output confusing you.
In fact, your metadata is already used up without available space.
GlobalReserve should also be counted as Metadata *used* space.
The following patch should make the output more meaningful.
[PATCH] btrfs-progs: Format change for btrfs fi df
>
> Even though the original pre-recovery crashed paritition showed 1.16GiB/2.01GiB
> metadata used, it appears used metadata can quickly grow.
>
> Deleting some subvolumes brought the metadata used down to 1.02GiB but:
>
> # btrfs balance start -dlimit=5 results in "ERROR: error during balancing '/' - No space left on device"
>
> Suggests that even with a used/total metadata of 1.02GiB/2.01GiB there are
> still issues. Perhaps the used portion can quickly grow to the point where it
> causes problems?
In fact the real used/total metadata is 1.52G/2.01G.
>
> Repeating balance again, starting with dlimit=1 then dlimit=2 then dlimit=5,
> was successful with 8 chunks reallocated and "btrfs fi show /" showing
> 103.02GiB/108.80GiB allocated.
Good, 5GiB freed space, it can be allocated for metadata to slightly
reduce the metadata pressure.
But not for long.
The root resolve will be, add more space into this btrfs.
Thanks,
Qu
>
> I have an image of the crashed partition so let me know if any further analysis
> would be useful. I would guess that the low free metadata may have been a
> factor in the crash; it was enough of an issue to cause the post-recovery
> system to be unable to boot properly.
>
> FWIW I found several other reports of unbootable btrfs with the same "corrupt
> leaf, slot offset bad" error, some also reporting that it occurred after a
> crash that required a hard reset:
>
> https://bugzilla.kernel.org/show_bug.cgi?id=74611
> http://unix.stackexchange.com/questions/195888/how-to-repair-a-btrfs-block
> http://www.spinics.net/lists/linux-btrfs/msg48167.html
> http://ubuntuforums.org/showthread.php?t=2301288
> https://lists.archlinux.org/pipermail/arch-general/2014-April/035885.html
> http://comments.gmane.org/gmane.comp.file-systems.btrfs/34190
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
>
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Crash, boot mount failure: "corrupt leaf, slot offset bad"
2016-01-05 1:57 ` Qu Wenruo
@ 2016-01-05 13:41 ` Chris Bainbridge
2016-01-06 0:57 ` Qu Wenruo
0 siblings, 1 reply; 6+ messages in thread
From: Chris Bainbridge @ 2016-01-05 13:41 UTC (permalink / raw)
To: Qu Wenruo; +Cc: linux-btrfs
On 5 January 2016 at 01:57, Qu Wenruo <quwenruo@cn.fujitsu.com> wrote:
>>
>> Data, single: total=106.79GiB, used=82.01GiB
>> System, single: total=4.00MiB, used=16.00KiB
>> Metadata, single: total=2.01GiB, used=1.51GiB
>> GlobalReserve, single: total=512.00MiB, used=0.00B
>
>
> That's the btrfs fi df misleading output confusing you.
>
> In fact, your metadata is already used up without available space.
> GlobalReserve should also be counted as Metadata *used* space.
Thanks for the explanation - the FAQ[1] misleads when it describes
GlobalReserve as "The block reserve is only virtual and is not stored
on the devices." - which sounds like the reserve is literally not
stored on the drive.
The FAQ[2] also suggests that the free space in metadata can be less
than the block reserve total:
"If the free space in metadata is less than or equal to the block
reserve value (typically 512 MiB, but might be something else on a
particularly small or large filesystem), then it's close to full."
But what you are saying is that this is wrong and the free space in
metadata can never be less than the block reserve, because the block
reserve includes the metadata free space?
[1] https://btrfs.wiki.kernel.org/index.php/FAQ#What_is_the_GlobalReserve_and_why_does_.27btrfs_fi_df.27_show_it_as_single_even_on_RAID_filesystems.3F
[2] https://btrfs.wiki.kernel.org/index.php/FAQ#if_your_device_is_large_.28.3E16GiB.29
> Good, 5GiB freed space, it can be allocated for metadata to slightly reduce
> the metadata pressure.
>
> But not for long.
> The root resolve will be, add more space into this btrfs.
Yes but this is a 128GB SSD and metadata could have been reallocated
from some of the 25GB of free space allocated to data. Even with a
bigger drive, it is possible that chunks could be allocated to data,
and then later operations requiring more metadata will still run out
(running out of metadata space seems to be a reasonably common
occurrence judging by the number of "why is btrfs reporting no space
when I have space free" questions). The file system shouldn't be
corrupted when that happens.
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Crash, boot mount failure: "corrupt leaf, slot offset bad"
2016-01-05 13:41 ` Chris Bainbridge
@ 2016-01-06 0:57 ` Qu Wenruo
2016-01-06 1:31 ` Chris Bainbridge
0 siblings, 1 reply; 6+ messages in thread
From: Qu Wenruo @ 2016-01-06 0:57 UTC (permalink / raw)
To: Chris Bainbridge; +Cc: linux-btrfs
Chris Bainbridge wrote on 2016/01/05 13:41 +0000:
> On 5 January 2016 at 01:57, Qu Wenruo <quwenruo@cn.fujitsu.com> wrote:
>>>
>>> Data, single: total=106.79GiB, used=82.01GiB
>>> System, single: total=4.00MiB, used=16.00KiB
>>> Metadata, single: total=2.01GiB, used=1.51GiB
>>> GlobalReserve, single: total=512.00MiB, used=0.00B
>>
>>
>> That's the btrfs fi df misleading output confusing you.
>>
>> In fact, your metadata is already used up without available space.
>> GlobalReserve should also be counted as Metadata *used* space.
>
> Thanks for the explanation - the FAQ[1] misleads when it describes
> GlobalReserve as "The block reserve is only virtual and is not stored
> on the devices." - which sounds like the reserve is literally not
> stored on the drive.
In fact FAQ description is not wrong either.
GlobalReserve is not stored in any where, that's true.
Since it doesn't takes space(unless its used is not 0), it is stored no
where and FAQ is right.
Metadata allocation algorithm will try its best to keep enough free
space for GlobalReserve.
So for end user, space you can't directly use is no different from used
space.
>
> The FAQ[2] also suggests that the free space in metadata can be less
> than the block reserve total:
>
> "If the free space in metadata is less than or equal to the block
> reserve value (typically 512 MiB, but might be something else on a
> particularly small or large filesystem), then it's close to full."
>
> But what you are saying is that this is wrong and the free space in
> metadata can never be less than the block reserve, because the block
> reserve includes the metadata free space?
Sorry for the confusion.
Yes, it's possible for available metadata space less than global reserve
space.
But when it happens, your used space in GlobalReserved is not 0, and
unfortunately you are already super short of space.
Meaning you are even unable to touch an empty file.
And in that case, if your kernel is not new enough, you can't even
delete a file thanks to the metadata COW.
So for common case, one can just treat global reserve as used metadata,
unless used global reserve is not 0.
>
> [1] https://btrfs.wiki.kernel.org/index.php/FAQ#What_is_the_GlobalReserve_and_why_does_.27btrfs_fi_df.27_show_it_as_single_even_on_RAID_filesystems.3F
> [2] https://btrfs.wiki.kernel.org/index.php/FAQ#if_your_device_is_large_.28.3E16GiB.29
>
>> Good, 5GiB freed space, it can be allocated for metadata to slightly reduce
>> the metadata pressure.
>>
>> But not for long.
>> The root resolve will be, add more space into this btrfs.
>
> Yes but this is a 128GB SSD and metadata could have been reallocated
> from some of the 25GB of free space allocated to data.
This can only happens when:
1) All data chunk is balanced into super compact case, to free all the 25G
Since btrfs store data and metadata into different chunks, one needs
to use balance to free space from allocated data/metadata chunks.
And in your case, you just tried dlimit=1 2 and 5, which will only
free at most 8 chunks (and at most 8G space).
If you want to free all the 25G free space from data chunks, then no
dlimit at all.
2) Mixed block groups.
This is the most straightforward case.
All data and metadata can be stored into the same chunk. Then no
such problem at all.
But developers tends to avoid such behavior though.
> Even with a
> bigger drive, it is possible that chunks could be allocated to data,
> and then later operations requiring more metadata will still run out
> (running out of metadata space seems to be a reasonably common
> occurrence judging by the number of "why is btrfs reporting no space
> when I have space free" questions).
This is true, and that's the long existing btrfs problem.
Except balance and add more devices, there is no super good ideas so far.
Maybe one day we can enhance it from the allocation algorithm.
> The file system shouldn't be corrupted when that happens.
>
I'm sorry that I'm off topic for the GlobalReserve and unbalanced
data/metadata chunk.
But I don't consider the corruption is caused by unbalanced
data/metadata chunks.
So let's go back to the corruption case.
Since you took the image of the corrupted fs, would you please try the
following commands on the corrupted fs?
$ btrfs-debug-tree -b 67239936 <dumped image>
And, what the kernel mount option for the fs before crash?
The kernel messages shows that your tree root is corrupted.
This is common for a power loss.
But the problem is, btrfs uses barrier to ensure superblock is written
to disk *after* all other metadata committed.
Or superblock is not updated and still points to old metadata, makes
everything fine.
So, either barrier is broken or you specified nobarrier, or the power
loss directly corrupted the new tree root and magically makes the csum
still match.
Thanks,
Qu
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Crash, boot mount failure: "corrupt leaf, slot offset bad"
2016-01-06 0:57 ` Qu Wenruo
@ 2016-01-06 1:31 ` Chris Bainbridge
2016-02-04 20:05 ` Chris Bainbridge
0 siblings, 1 reply; 6+ messages in thread
From: Chris Bainbridge @ 2016-01-06 1:31 UTC (permalink / raw)
To: Qu Wenruo; +Cc: linux-btrfs
On Wed, Jan 06, 2016 at 08:57:28AM +0800, Qu Wenruo wrote:
>
> Since you took the image of the corrupted fs, would you please try the
> following commands on the corrupted fs?
>
> $ btrfs-debug-tree -b 67239936 <dumped image>
Command runs then segfaults:
leaf 67239936 items 92 free space 9138 generation 276688 owner 2
fs uuid b1103526-98a3-4b40-a782-cf66721ed600
chunk uuid 16e767e3-a321-4d0f-9c72-6ebac9d305c4
item 0 key (61513990144 EXTENT_ITEM 16384) itemoff 16232 itemsize 51
extent refs 1 gen 276685 flags TREE_BLOCK
tree block key (61617864704 EXTENT_ITEM 16384) level 0
tree block backref root 2
item 1 key (61514006528 EXTENT_ITEM 16384) itemoff 16181 itemsize 51
extent refs 1 gen 276285 flags TREE_BLOCK
tree block key (27627 DIR_INDEX 3576) level 0
tree block backref root 260
item 2 key (61514022912 EXTENT_ITEM 16384) itemoff 16130 itemsize 51
extent refs 1 gen 266026 flags TREE_BLOCK
tree block key (EXTENT_CSUM EXTENT_CSUM 99424354304) level 0
tree block backref root 7
item 3 key (61514039296 EXTENT_ITEM 16384) itemoff 16079 itemsize 51
extent refs 1 gen 275904 flags TREE_BLOCK
tree block key (1555518 INODE_ITEM 0) level 0
tree block backref root 260
item 4 key (61514055680 EXTENT_ITEM 16384) itemoff 16028 itemsize 51
extent refs 1 gen 276685 flags TREE_BLOCK
tree block key (61702111232 EXTENT_ITEM 16384) level 0
tree block backref root 2
item 5 key (61514088448 EXTENT_ITEM 16384) itemoff 15977 itemsize 51
extent refs 1 gen 276285 flags TREE_BLOCK
tree block key (1906872 INODE_REF 34185) level 0
tree block backref root 260
item 6 key (61514104832 EXTENT_ITEM 16384) itemoff 15926 itemsize 51
extent refs 1 gen 276685 flags TREE_BLOCK
tree block key (61741957120 EXTENT_ITEM 16384) level 0
tree block backref root 2
item 7 key (61514121216 EXTENT_ITEM 16384) itemoff 15875 itemsize 51
extent refs 1 gen 266026 flags TREE_BLOCK
tree block key (EXTENT_CSUM EXTENT_CSUM 99654656000) level 0
tree block backref root 7
item 8 key (61514137600 EXTENT_ITEM 16384) itemoff 15824 itemsize 51
extent refs 1 gen 266026 flags TREE_BLOCK
tree block key (EXTENT_CSUM EXTENT_CSUM 99620417536) level 0
tree block backref root 7
item 9 key (61514153984 EXTENT_ITEM 16384) itemoff 15773 itemsize 51
extent refs 1 gen 266026 flags TREE_BLOCK
tree block key (EXTENT_CSUM EXTENT_CSUM 99669962752) level 0
tree block backref root 7
item 10 key (61514170368 EXTENT_ITEM 16384) itemoff 15722 itemsize 51
extent refs 1 gen 266026 flags TREE_BLOCK
tree block key (EXTENT_CSUM EXTENT_CSUM 99639615488) level 0
tree block backref root 7
item 11 key (61514186752 EXTENT_ITEM 16384) itemoff 15671 itemsize 51
extent refs 1 gen 266026 flags TREE_BLOCK
tree block key (EXTENT_CSUM EXTENT_CSUM 99681320960) level 0
tree block backref root 7
item 12 key (61514203136 EXTENT_ITEM 16384) itemoff 15620 itemsize 51
extent refs 1 gen 276285 flags TREE_BLOCK
tree block key (882130 INODE_ITEM 0) level 0
tree block backref root 260
item 13 key (61514219520 EXTENT_ITEM 16384) itemoff 15569 itemsize 51
extent refs 1 gen 276685 flags TREE_BLOCK
tree block key (61831168000 EXTENT_ITEM 16384) level 0
tree block backref root 2
item 14 key (61514268672 EXTENT_ITEM 16384) itemoff 15518 itemsize 51
extent refs 1 gen 275904 flags TREE_BLOCK
tree block key (1553336 INODE_ITEM 0) level 0
tree block backref root 260
item 15 key (61514285056 EXTENT_ITEM 16384) itemoff 15467 itemsize 51
extent refs 1 gen 276685 flags TREE_BLOCK
tree block key (62053400576 EXTENT_ITEM 16384) level 0
tree block backref root 2
item 16 key (61514334208 EXTENT_ITEM 16384) itemoff 15416 itemsize 51
extent refs 1 gen 266026 flags TREE_BLOCK
tree block key (EXTENT_CSUM EXTENT_CSUM 99928444928) level 0
tree block backref root 7
item 17 key (61514350592 EXTENT_ITEM 16384) itemoff 15365 itemsize 51
extent refs 1 gen 266026 flags TREE_BLOCK
tree block key (EXTENT_CSUM EXTENT_CSUM 99940794368) level 0
tree block backref root 7
item 18 key (61514366976 EXTENT_ITEM 16384) itemoff 15314 itemsize 51
extent refs 1 gen 274438 flags TREE_BLOCK
tree block key (1532894 EXTENT_DATA 0) level 1
tree block backref root 3098
item 19 key (61514383360 EXTENT_ITEM 16384) itemoff 15263 itemsize 51
extent refs 1 gen 258313 flags TREE_BLOCK
tree block key (EXTENT_CSUM EXTENT_CSUM 121949413376) level 0
tree block backref root 7
item 20 key (61514399744 EXTENT_ITEM 16384) itemoff 15212 itemsize 51
extent refs 1 gen 274438 flags TREE_BLOCK
tree block key (1535799 INODE_ITEM 0) level 0
tree block backref root 3098
item 21 key (61514416128 EXTENT_ITEM 16384) itemoff 15161 itemsize 51
extent refs 1 gen 276685 flags TREE_BLOCK
tree block key (62136385536 EXTENT_ITEM 16384) level 0
tree block backref root 2
item 22 key (61514432512 EXTENT_ITEM 16384) itemoff 15110 itemsize 51
extent refs 1 gen 276685 flags TREE_BLOCK
tree block key (62174904320 EXTENT_ITEM 16384) level 0
tree block backref root 2
item 23 key (61514448896 EXTENT_ITEM 16384) itemoff 15059 itemsize 51
extent refs 1 gen 266026 flags TREE_BLOCK
tree block key (EXTENT_CSUM EXTENT_CSUM 99996536832) level 0
tree block backref root 7
item 24 key (61514465280 EXTENT_ITEM 16384) itemoff 15008 itemsize 51
extent refs 1 gen 266026 flags TREE_BLOCK
tree block key (EXTENT_CSUM EXTENT_CSUM 100002021376) level 0
tree block backref root 7
item 25 key (61514481664 EXTENT_ITEM 16384) itemoff 14957 itemsize 51
extent refs 1 gen 276285 flags TREE_BLOCK
tree block key (115779 DIR_ITEM 305173055) level 0
tree block backref root 260
item 26 key (61514498048 EXTENT_ITEM 16384) itemoff 14906 itemsize 51
extent refs 1 gen 276685 flags TREE_BLOCK
tree block key (63221645312 EXTENT_ITEM 28672) level 0
tree block backref root 2
item 27 key (61514547200 EXTENT_ITEM 16384) itemoff 14855 itemsize 51
extent refs 1 gen 266026 flags TREE_BLOCK
tree block key (EXTENT_CSUM EXTENT_CSUM 100013899776) level 0
tree block backref root 7
item 28 key (61514579968 EXTENT_ITEM 16384) itemoff 14804 itemsize 51
extent refs 1 gen 266026 flags TREE_BLOCK
tree block key (EXTENT_CSUM EXTENT_CSUM 100037701632) level 0
tree block backref root 7
item 29 key (61514596352 EXTENT_ITEM 16384) itemoff 14753 itemsize 51
extent refs 1 gen 276285 flags TREE_BLOCK
tree block key (1902695 INODE_ITEM 0) level 0
tree block backref root 260
item 30 key (61514629120 EXTENT_ITEM 16384) itemoff 14702 itemsize 51
extent refs 1 gen 276685 flags TREE_BLOCK
tree block key (65896304640 EXTENT_ITEM 16384) level 0
tree block backref root 2
item 31 key (61514645504 EXTENT_ITEM 16384) itemoff 14651 itemsize 51
extent refs 1 gen 266026 flags TREE_BLOCK
tree block key (EXTENT_CSUM EXTENT_CSUM 100167794688) level 0
tree block backref root 7
item 32 key (61514661888 EXTENT_ITEM 16384) itemoff 14600 itemsize 51
extent refs 1 gen 276285 flags TREE_BLOCK
tree block key (884400 INODE_ITEM 0) level 0
tree block backref root 260
item 33 key (61514678272 EXTENT_ITEM 16384) itemoff 14549 itemsize 51
extent refs 1 gen 276685 flags TREE_BLOCK
tree block key (65969061888 EXTENT_ITEM 20480) level 0
tree block backref root 2
item 34 key (61514694656 EXTENT_ITEM 16384) itemoff 14498 itemsize 51
extent refs 1 gen 276685 flags TREE_BLOCK
tree block key (65975934976 EXTENT_ITEM 8192) level 0
tree block backref root 2
item 35 key (61514711040 EXTENT_ITEM 16384) itemoff 14447 itemsize 51
extent refs 1 gen 266026 flags TREE_BLOCK
tree block key (EXTENT_CSUM EXTENT_CSUM 100226453504) level 0
tree block backref root 7
item 36 key (61514727424 EXTENT_ITEM 16384) itemoff 14396 itemsize 51
extent refs 1 gen 276285 flags TREE_BLOCK
tree block key (27641 DIR_INDEX 3636) level 0
tree block backref root 260
item 37 key (61514743808 EXTENT_ITEM 16384) itemoff 14345 itemsize 51
extent refs 1 gen 276285 flags TREE_BLOCK
tree block key (115779 DIR_INDEX 13840) level 0
tree block backref root 260
item 38 key (61514760192 EXTENT_ITEM 16384) itemoff 14294 itemsize 51
extent refs 1 gen 266026 flags TREE_BLOCK
tree block key (EXTENT_CSUM EXTENT_CSUM 100303273984) level 0
tree block backref root 7
item 39 key (61514776576 EXTENT_ITEM 16384) itemoff 14243 itemsize 51
extent refs 1 gen 266026 flags TREE_BLOCK
tree block key (EXTENT_CSUM EXTENT_CSUM 100324823040) level 0
tree block backref root 7
item 40 key (61514792960 EXTENT_ITEM 16384) itemoff 14192 itemsize 51
extent refs 1 gen 266026 flags TREE_BLOCK
tree block key (EXTENT_CSUM EXTENT_CSUM 100288327680) level 0
tree block backref root 7
item 41 key (61514809344 EXTENT_ITEM 16384) itemoff 14141 itemsize 51
extent refs 1 gen 276285 flags TREE_BLOCK
tree block key (115778 DIR_INDEX 15836) level 0
tree block backref root 260
item 42 key (61514825728 EXTENT_ITEM 16384) itemoff 14072 itemsize 69
extent refs 3 gen 270120 flags TREE_BLOCK|FULL_BACKREF
tree block key (1468220 INODE_ITEM 0) level 0
shared block backref parent 61683843072
shared block backref parent 743489536
shared block backref parent 121651200
item 43 key (61514842112 EXTENT_ITEM 16384) itemoff 13994 itemsize 78
extent refs 4 gen 270120 flags TREE_BLOCK
tree block key (1468243 EXTENT_DATA 53248) level 0
tree block backref root 257
shared block backref parent 61683843072
shared block backref parent 743489536
shared block backref parent 121651200
item 44 key (61514858496 EXTENT_ITEM 16384) itemoff 13943 itemsize 51
extent refs 1 gen 266026 flags TREE_BLOCK
tree block key (EXTENT_CSUM EXTENT_CSUM 100277510144) level 0
tree block backref root 7
item 45 key (61514874880 EXTENT_ITEM 16384) itemoff 13892 itemsize 51
extent refs 1 gen 276685 flags TREE_BLOCK
tree block key (66766774272 EXTENT_ITEM 4096) level 0
tree block backref root 2
item 46 key (61514891264 EXTENT_ITEM 16384) itemoff 13841 itemsize 51
extent refs 1 gen 276685 flags TREE_BLOCK
tree block key (66770665472 EXTENT_ITEM 4096) level 0
tree block backref root 2
item 47 key (61514907648 EXTENT_ITEM 16384) itemoff 13790 itemsize 51
extent refs 1 gen 276685 flags TREE_BLOCK
tree block key (66785505280 EXTENT_ITEM 4096) level 0
tree block backref root 2
item 48 key (61514924032 EXTENT_ITEM 16384) itemoff 13739 itemsize 51
extent refs 1 gen 266026 flags TREE_BLOCK
tree block key (EXTENT_CSUM EXTENT_CSUM 100388798464) level 0
tree block backref root 7
item 49 key (61514940416 EXTENT_ITEM 16384) itemoff 13688 itemsize 51
extent refs 1 gen 266026 flags TREE_BLOCK
tree block key (EXTENT_CSUM EXTENT_CSUM 100400062464) level 0
tree block backref root 7
item 50 key (61514956800 EXTENT_ITEM 16384) itemoff 13637 itemsize 51
extent refs 1 gen 276685 flags TREE_BLOCK
tree block key (66799345664 EXTENT_ITEM 16384) level 0
tree block backref root 2
item 51 key (61514973184 EXTENT_ITEM 16384) itemoff 13586 itemsize 51
extent refs 1 gen 266026 flags TREE_BLOCK
tree block key (EXTENT_CSUM EXTENT_CSUM 100672151552) level 0
tree block backref root 7
item 52 key (61514989568 EXTENT_ITEM 16384) itemoff 13535 itemsize 51
extent refs 1 gen 276685 flags TREE_BLOCK
tree block key (66819731456 EXTENT_ITEM 4096) level 0
tree block backref root 2
item 53 key (61515005952 EXTENT_ITEM 16384) itemoff 13484 itemsize 51
extent refs 1 gen 276685 flags TREE_BLOCK
tree block key (67328659456 EXTENT_ITEM 4096) level 0
tree block backref root 2
item 54 key (61515022336 EXTENT_ITEM 16384) itemoff 13433 itemsize 51
extent refs 1 gen 276685 flags TREE_BLOCK
tree block key (68718272512 EXTENT_ITEM 135168) level 0
tree block backref root 2
item 55 key (61515038720 EXTENT_ITEM 16384) itemoff 13382 itemsize 51
extent refs 1 gen 266026 flags TREE_BLOCK
tree block key (EXTENT_CSUM EXTENT_CSUM 100768583680) level 0
tree block backref root 7
item 56 key (61515055104 EXTENT_ITEM 16384) itemoff 13331 itemsize 51
extent refs 1 gen 266026 flags TREE_BLOCK
tree block key (EXTENT_CSUM EXTENT_CSUM 100871962624) level 0
tree block backref root 7
item 57 key (61515071488 EXTENT_ITEM 16384) itemoff 13262 itemsize 69
extent refs 3 gen 270120 flags TREE_BLOCK|FULL_BACKREF
tree block key (1461920 INODE_ITEM 0) level 0
shared block backref parent 61683843072
shared block backref parent 743489536
shared block backref parent 121651200
item 58 key (61515087872 EXTENT_ITEM 16384) itemoff 13193 itemsize 69
extent refs 3 gen 270120 flags TREE_BLOCK|FULL_BACKREF
tree block key (1327416 INODE_ITEM 0) level 0
shared block backref parent 62029692928
shared block backref parent 1062486016
shared block backref parent 699891712
item 59 key (61515120640 EXTENT_ITEM 16384) itemoff 13115 itemsize 78
extent refs 4 gen 262494 flags TREE_BLOCK
tree block key (1419842 INODE_ITEM 0) level 0
tree block backref root 257
shared block backref parent 61683843072
shared block backref parent 730775552
shared block backref parent 121651200
item 60 key (61515137024 EXTENT_ITEM 16384) itemoff 13037 itemsize 78
extent refs 4 gen 270120 flags TREE_BLOCK
tree block key (1192152 INODE_REF 7310) level 0
tree block backref root 257
shared block backref parent 61548216320
shared block backref parent 1087340544
shared block backref parent 748519424
item 61 key (61515153408 EXTENT_ITEM 16384) itemoff 12986 itemsize 51
extent refs 1 gen 276285 flags TREE_BLOCK
tree block key (35908 DIR_ITEM 1968783234) level 0
tree block backref root 260
item 62 key (61515169792 EXTENT_ITEM 16384) itemoff 12935 itemsize 51
extent refs 1 gen 276685 flags TREE_BLOCK
tree block key (70525399040 EXTENT_ITEM 8192) level 0
tree block backref root 2
item 63 key (61515186176 EXTENT_ITEM 16384) itemoff 12884 itemsize 51
extent refs 1 gen 266026 flags TREE_BLOCK
tree block key (EXTENT_CSUM EXTENT_CSUM 100904398848) level 0
tree block backref root 7
item 64 key (61515202560 EXTENT_ITEM 16384) itemoff 12815 itemsize 69
extent refs 3 gen 270120 flags TREE_BLOCK|FULL_BACKREF
tree block key (1057465 INODE_ITEM 0) level 0
shared block backref parent 61506486272
shared block backref parent 970948608
shared block backref parent 768196608
item 65 key (61515218944 EXTENT_ITEM 16384) itemoff 12764 itemsize 51
extent refs 1 gen 276685 flags TREE_BLOCK
tree block key (70638366720 EXTENT_ITEM 40960) level 0
tree block backref root 2
item 66 key (61515235328 EXTENT_ITEM 16384) itemoff 12713 itemsize 51
extent refs 1 gen 276685 flags TREE_BLOCK
tree block key (70894116864 EXTENT_ITEM 12288) level 0
tree block backref root 2
item 67 key (61515251712 EXTENT_ITEM 16384) itemoff 12662 itemsize 51
extent refs 1 gen 266026 flags TREE_BLOCK
tree block key (EXTENT_CSUM EXTENT_CSUM 100967411712) level 0
tree block backref root 7
item 68 key (61515268096 EXTENT_ITEM 16384) itemoff 12611 itemsize 51
extent refs 1 gen 276685 flags TREE_BLOCK
tree block key (70952181760 EXTENT_ITEM 40960) level 0
tree block backref root 2
item 69 key (61515284480 EXTENT_ITEM 16384) itemoff 12560 itemsize 51
extent refs 1 gen 266026 flags TREE_BLOCK
tree block key (EXTENT_CSUM EXTENT_CSUM 100994502656) level 0
tree block backref root 7
item 70 key (61515317248 EXTENT_ITEM 16384) itemoff 12509 itemsize 51
extent refs 1 gen 266026 flags TREE_BLOCK
tree block key (EXTENT_CSUM EXTENT_CSUM 101000675328) level 0
tree block backref root 7
item 71 key (61515333632 EXTENT_ITEM 16384) itemoff 12458 itemsize 51
extent refs 1 gen 266026 flags TREE_BLOCK
tree block key (EXTENT_CSUM EXTENT_CSUM 101020729344) level 0
tree block backref root 7
item 72 key (61515350016 EXTENT_ITEM 16384) itemoff 12407 itemsize 51
extent refs 1 gen 276685 flags TREE_BLOCK
tree block key (74755063808 EXTENT_ITEM 8192) level 0
tree block backref root 2
item 73 key (61515382784 EXTENT_ITEM 16384) itemoff 12356 itemsize 51
extent refs 1 gen 276285 flags TREE_BLOCK
tree block key (1953118 INODE_REF 19532) level 0
tree block backref root 260
item 74 key (61515399168 EXTENT_ITEM 16384) itemoff 12305 itemsize 51
extent refs 1 gen 266026 flags TREE_BLOCK
tree block key (EXTENT_CSUM EXTENT_CSUM 101086830592) level 0
tree block backref root 7
item 75 key (61515415552 EXTENT_ITEM 16384) itemoff 12254 itemsize 51
extent refs 1 gen 276685 flags TREE_BLOCK
tree block key (75363184640 EXTENT_ITEM 4096) level 0
tree block backref root 2
item 76 key (61515431936 EXTENT_ITEM 16384) itemoff 12203 itemsize 51
extent refs 1 gen 266026 flags TREE_BLOCK
tree block key (EXTENT_CSUM EXTENT_CSUM 101246300160) level 0
tree block backref root 7
item 77 key (61515448320 EXTENT_ITEM 16384) itemoff 12152 itemsize 51
extent refs 1 gen 266026 flags TREE_BLOCK
tree block key (EXTENT_CSUM EXTENT_CSUM 101663543296) level 0
tree block backref root 7
item 78 key (61515464704 EXTENT_ITEM 16384) itemoff 12101 itemsize 51
extent refs 1 gen 276685 flags TREE_BLOCK
tree block key (75372187648 EXTENT_ITEM 1097728) level 0
tree block backref root 2
item 79 key (61515481088 EXTENT_ITEM 16384) itemoff 12050 itemsize 51
extent refs 1 gen 276685 flags TREE_BLOCK
tree block key (75427885056 EXTENT_ITEM 20480) level 0
tree block backref root 2
item 80 key (61515497472 EXTENT_ITEM 16384) itemoff 11999 itemsize 51
extent refs 1 gen 276685 flags TREE_BLOCK
tree block key (75457257472 EXTENT_ITEM 8192) level 0
tree block backref root 2
item 81 key (61515513856 EXTENT_ITEM 16384) itemoff 11948 itemsize 51
extent refs 1 gen 276685 flags TREE_BLOCK
tree block key (76215603200 EXTENT_ITEM 4096) level 0
tree block backref root 2
item 82 key (61515530240 EXTENT_ITEM 16384) itemoff 11897 itemsize 51
extent refs 1 gen 276685 flags TREE_BLOCK
tree block key (76529934336 EXTENT_ITEM 12288) level 0
tree block backref root 2
item 83 key (61515546624 EXTENT_ITEM 16384) itemoff 5451334 itemsize 262144
Program received signal SIGSEGV, Segmentation fault.
0x000000000041b5c9 in btrfs_extent_flags (s=0x532eab, eb=0x672c30) at ctree.h:1511
1511 BTRFS_SETGET_FUNCS(extent_flags, struct btrfs_extent_item, flags, 64);
(gdb) bt
#0 0x000000000041b5c9 in btrfs_extent_flags (s=0x532eab, eb=0x672c30) at ctree.h:1511
#1 print_extent_item (eb=eb@entry=0x672c30, slot=slot@entry=83, metadata=metadata@entry=0) at print-tree.c:374
#2 0x000000000041c1cd in btrfs_print_leaf (root=root@entry=0x6699a0, l=l@entry=0x672c30) at print-tree.c:903
#3 0x000000000041c75b in btrfs_print_tree (root=root@entry=0x6699a0, eb=eb@entry=0x672c30, follow=follow@entry=0) at print-tree.c:1065
#4 0x00000000004355d2 in main (ac=<optimised out>, av=<optimised out>) at btrfs-debug-tree.c:232
> And, what the kernel mount option for the fs before crash?
Nothing unusual, just defaults and noatime
mount shows: rw,noatime,ssd,noacl,space_cache
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Crash, boot mount failure: "corrupt leaf, slot offset bad"
2016-01-06 1:31 ` Chris Bainbridge
@ 2016-02-04 20:05 ` Chris Bainbridge
0 siblings, 0 replies; 6+ messages in thread
From: Chris Bainbridge @ 2016-02-04 20:05 UTC (permalink / raw)
To: Qu Wenruo; +Cc: linux-btrfs
I had another btrfs crash with identical symptoms. This time I managed
to reproduce the crash and identify the root cause. It turns out that
the Apple boot firmware is buggy and leaves wifi DMA turned on, which
randomly corrupts memory after Linux is running.
https://bugzilla.kernel.org/show_bug.cgi?id=111781
So btrfs is not to blame after all.
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2016-02-04 20:06 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2016-01-04 17:05 Crash, boot mount failure: "corrupt leaf, slot offset bad" Chris Bainbridge
2016-01-05 1:57 ` Qu Wenruo
2016-01-05 13:41 ` Chris Bainbridge
2016-01-06 0:57 ` Qu Wenruo
2016-01-06 1:31 ` Chris Bainbridge
2016-02-04 20:05 ` Chris Bainbridge
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).