* Errors found in extent allocation tree (single bit flipped)
@ 2024-08-12 14:57 Pieter P
2024-08-20 14:01 ` Pieter P
2024-08-20 23:02 ` Qu Wenruo
0 siblings, 2 replies; 12+ messages in thread
From: Pieter P @ 2024-08-12 14:57 UTC (permalink / raw)
To: linux-btrfs
[-- Attachment #1: Type: text/plain, Size: 2198 bytes --]
Hello,
My btrfs partition suddenly went read-only three different times over
the past couple of days. Looking at the dmesg output (see attachment),
and then running btrfs check, it seems that a single bit is flipped
somewhere in a btrfs bytenr (2216718336 = 0x84207000, 576460754520141824
= 0x800000084207000).
I've verified that my RAM, SSD and other hardware are fine using Dell's
preboot diagnostic tool and Samsung's SSD magician full diagnostic scan.
Can this problem be fixed by running btrfs check --repair from a live
USB? (I'm using Ubuntu 22.04 with btrfs-progs v5.16.2).
The issue seems similar to
https://lore.kernel.org/linux-btrfs/o9XJJu0qMzbW4Iiu4eBNXZZscccGRCC0lpf_tUMBA7Mlxmlba4yfx27_7W29DfGvZAByVGSyul3dxcPwpLrPBWiUx0B-AJU-L-vyKKMyjO8=@proton.me/#r,
but the output from btrfs check is different.
Thank you!
Pieter
sudo btrfs check --readonly /dev/mapper/luks-xxxxx
Opening filesystem to check...
Checking filesystem on /dev/mapper/luks-xxxxx
UUID: xxxxx
[1/7] checking root items
[2/7] checking extents
ref mismatch on [2216718336 4096] extent item 1, found 0
incorrect local backref count on 2216718336 root 257 owner 50058751
offset 0 found 0 wanted 1 back 0x5aef8c56e010
backref disk bytenr does not match extent record, bytenr=2216718336, ref
bytenr=0
backpointer mismatch on [2216718336 4096]
owner ref check failed [2216718336 4096]
ref mismatch on [576460754520141824 4096] extent item 0, found 1
data backref 576460754520141824 root 257 owner 50058751 offset 0
num_refs 0 not found in extent tree
incorrect local backref count on 576460754520141824 root 257 owner
50058751 offset 0 found 1 wanted 0 back 0x5aefa0858d50
backpointer mismatch on [576460754520141824 4096]
ERROR: errors found in extent allocation tree or chunk allocation
[3/7] checking free space tree
[4/7] checking fs roots
root 257 inode 50058751 errors 1000, some csum missing
ERROR: errors found in fs roots
found 598926008327 bytes used, error(s) found
total csum bytes: 565741960
total tree bytes: 14748745728
total fs tree bytes: 13102383104
total extent tree bytes: 908722176
btree space waste bytes: 2521891839
file data blocks allocated: 824444620800
referenced 1059369144320
[-- Attachment #2: dmesg.txt --]
[-- Type: text/plain, Size: 30087 bytes --]
[aug12 14:28] ------------[ cut here ]------------
[ +0,000003] WARNING: CPU: 4 PID: 894 at fs/btrfs/extent-tree.c:3112 __btrfs_free_extent+0x603/0xf10 [btrfs]
[ +0,000101] Modules linked in: snd_seq_midi snd_seq_midi_event snd_seq snd_rawmidi snd_seq_device snd_soc_avs snd_soc_hda_codec snd_hda_ext_core snd_soc_core snd_compress ac97_bus snd_pcm_dmaengine snd_ctl_led tls ses enclosure scsi_transport_sas rfcomm xt_conntrack nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack_netlink nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xfrm_user xfrm_algo xt_addrtype nft_compat nf_tables nfnetlink br_netfilter bridge stp llc ccm cmac algif_hash algif_skcipher af_alg bnep overlay snd_hda_codec_hdmi binfmt_misc snd_hda_codec_realtek snd_hda_codec_generic intel_tcc_cooling x86_pkg_temp_thermal snd_hda_intel intel_powerclamp snd_intel_dspcfg mei_hdcp mei_pxp snd_intel_sdw_acpi coretemp snd_hda_codec intel_rapl_msr snd_hda_core snd_hwdep dell_laptop snd_pcm kvm_intel dell_smm_hwmon kvm irqbypass rapl intel_cstate dell_wmi nls_iso8859_1 dell_smbios ledtrig_audio iwlmvm wmi_bmof mac80211 cdc_acm dcdbas uvcvideo joydev dell_wmi_descriptor input_leds videobuf2_vmalloc snd_timer libarc4 btusb btrtl
[ +0,000056] uvc btbcm mxm_wmi btintel serio_raw videobuf2_memops btmtk videobuf2_v4l2 intel_wmi_thunderbolt snd iwlwifi videodev soundcore ee1004 videobuf2_common cfg80211 bluetooth mc processor_thermal_device_pci_legacy hid_multitouch processor_thermal_device ecdh_generic ecc processor_thermal_rfim processor_thermal_mbox processor_thermal_rapl mei_me intel_rapl_common mei intel_pch_thermal intel_soc_dts_iosf int3403_thermal intel_hid int3400_thermal mac_hid acpi_pad int340x_thermal_zone sparse_keymap acpi_thermal_rel nvidia_uvm(PO) sch_fq_codel msr parport_pc ppdev lp parport efi_pstore ip_tables x_tables autofs4 btrfs blake2b_generic xor raid6_pq libcrc32c dm_crypt hid_logitech_hidpp hid_logitech_dj uas usb_storage hid_logitech nvidia_drm(PO) ff_memless nvidia_modeset(PO) nvidia(PO) usbhid i915 drm_buddy i2c_algo_bit ttm drm_display_helper crct10dif_pclmul crc32_pclmul polyval_clmulni cec polyval_generic ghash_clmulni_intel sha256_ssse3 rc_core sha1_ssse3 hid_generic aesni_intel drm_kms_helper nvme i2c_i801
[ +0,000058] crypto_simd rtsx_pci_sdmmc psmouse cryptd drm i2c_smbus nvme_core intel_lpss_pci nvme_common rtsx_pci intel_lpss ahci i2c_hid_acpi xhci_pci libahci i2c_hid xhci_pci_renesas idma64 hid video wmi [last unloaded: snd_seq_device]
[ +0,000017] CPU: 4 PID: 894 Comm: btrfs-transacti Tainted: P W O 6.5.0-45-generic #45~22.04.1-Ubuntu
[ +0,000002] Hardware name: Dell Inc. XPS 15 9560
[ +0,000002] RIP: 0010:__btrfs_free_extent+0x603/0xf10 [btrfs]
[ +0,000061] Code: 8b 3f e8 50 6f 00 00 49 8b 7e 50 45 8b 4f 40 49 89 d8 48 8b 4d a8 8b 55 10 48 c7 c6 70 ed 1d c4 e8 a2 02 0e 00 e9 9e fd ff ff <0f> 0b f0 48 0f ba 28 03 41 b8 00 00 00 00 0f 83 8a 03 00 00 b9 8b
[ +0,000002] RSP: 0018:ffffafda0052fb70 EFLAGS: 00010246
[ +0,000002] RAX: ffff92f701e449f8 RBX: 0800000084207000 RCX: 0000000000000000
[ +0,000002] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
[ +0,000001] RBP: ffffafda0052fc10 R08: 0000000000000000 R09: 0000000000000000
[ +0,000001] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000001
[ +0,000001] R13: ffff92fa2c3b0d20 R14: ffff92fb936e9c30 R15: ffff92fa407e2770
[ +0,000002] FS: 0000000000000000(0000) GS:ffff92fe7e500000(0000) knlGS:0000000000000000
[ +0,000001] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ +0,000002] CR2: 000000c00053b000 CR3: 00000003b8c94005 CR4: 00000000003706e0
[ +0,000001] Call Trace:
[ +0,000002] <TASK>
[ +0,000002] ? show_regs+0x6d/0x80
[ +0,000006] ? __warn+0x89/0x160
[ +0,000004] ? __btrfs_free_extent+0x603/0xf10 [btrfs]
[ +0,000060] ? report_bug+0x17e/0x1b0
[ +0,000005] ? handle_bug+0x46/0x90
[ +0,000003] ? exc_invalid_op+0x18/0x80
[ +0,000003] ? asm_exc_invalid_op+0x1b/0x20
[ +0,000004] ? __btrfs_free_extent+0x603/0xf10 [btrfs]
[ +0,000061] run_delayed_data_ref+0x8b/0x180 [btrfs]
[ +0,000058] btrfs_run_delayed_refs_for_head+0x183/0x550 [btrfs]
[ +0,000059] __btrfs_run_delayed_refs+0xc0/0x150 [btrfs]
[ +0,000058] btrfs_run_delayed_refs+0x6d/0x1f0 [btrfs]
[ +0,000058] btrfs_start_dirty_block_groups+0x36b/0x530 [btrfs]
[ +0,000063] btrfs_commit_transaction+0xb3/0xbb0 [btrfs]
[ +0,000064] ? start_transaction+0xcf/0x660 [btrfs]
[ +0,000063] transaction_kthread+0x14b/0x1c0 [btrfs]
[ +0,000063] ? __pfx_transaction_kthread+0x10/0x10 [btrfs]
[ +0,000061] kthread+0xef/0x120
[ +0,000004] ? __pfx_kthread+0x10/0x10
[ +0,000003] ret_from_fork+0x44/0x70
[ +0,000003] ? __pfx_kthread+0x10/0x10
[ +0,000002] ret_from_fork_asm+0x1b/0x30
[ +0,000004] </TASK>
[ +0,000001] ---[ end trace 0000000000000000 ]---
[ +0,000002] ------------[ cut here ]------------
[ +0,000001] BTRFS: Transaction aborted (error -117)
[ +0,000033] WARNING: CPU: 4 PID: 894 at fs/btrfs/extent-tree.c:3113 __btrfs_free_extent+0xd83/0xf10 [btrfs]
[ +0,000061] Modules linked in: snd_seq_midi snd_seq_midi_event snd_seq snd_rawmidi snd_seq_device snd_soc_avs snd_soc_hda_codec snd_hda_ext_core snd_soc_core snd_compress ac97_bus snd_pcm_dmaengine snd_ctl_led tls ses enclosure scsi_transport_sas rfcomm xt_conntrack nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack_netlink nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xfrm_user xfrm_algo xt_addrtype nft_compat nf_tables nfnetlink br_netfilter bridge stp llc ccm cmac algif_hash algif_skcipher af_alg bnep overlay snd_hda_codec_hdmi binfmt_misc snd_hda_codec_realtek snd_hda_codec_generic intel_tcc_cooling x86_pkg_temp_thermal snd_hda_intel intel_powerclamp snd_intel_dspcfg mei_hdcp mei_pxp snd_intel_sdw_acpi coretemp snd_hda_codec intel_rapl_msr snd_hda_core snd_hwdep dell_laptop snd_pcm kvm_intel dell_smm_hwmon kvm irqbypass rapl intel_cstate dell_wmi nls_iso8859_1 dell_smbios ledtrig_audio iwlmvm wmi_bmof mac80211 cdc_acm dcdbas uvcvideo joydev dell_wmi_descriptor input_leds videobuf2_vmalloc snd_timer libarc4 btusb btrtl
[ +0,000051] uvc btbcm mxm_wmi btintel serio_raw videobuf2_memops btmtk videobuf2_v4l2 intel_wmi_thunderbolt snd iwlwifi videodev soundcore ee1004 videobuf2_common cfg80211 bluetooth mc processor_thermal_device_pci_legacy hid_multitouch processor_thermal_device ecdh_generic ecc processor_thermal_rfim processor_thermal_mbox processor_thermal_rapl mei_me intel_rapl_common mei intel_pch_thermal intel_soc_dts_iosf int3403_thermal intel_hid int3400_thermal mac_hid acpi_pad int340x_thermal_zone sparse_keymap acpi_thermal_rel nvidia_uvm(PO) sch_fq_codel msr parport_pc ppdev lp parport efi_pstore ip_tables x_tables autofs4 btrfs blake2b_generic xor raid6_pq libcrc32c dm_crypt hid_logitech_hidpp hid_logitech_dj uas usb_storage hid_logitech nvidia_drm(PO) ff_memless nvidia_modeset(PO) nvidia(PO) usbhid i915 drm_buddy i2c_algo_bit ttm drm_display_helper crct10dif_pclmul crc32_pclmul polyval_clmulni cec polyval_generic ghash_clmulni_intel sha256_ssse3 rc_core sha1_ssse3 hid_generic aesni_intel drm_kms_helper nvme i2c_i801
[ +0,000049] crypto_simd rtsx_pci_sdmmc psmouse cryptd drm i2c_smbus nvme_core intel_lpss_pci nvme_common rtsx_pci intel_lpss ahci i2c_hid_acpi xhci_pci libahci i2c_hid xhci_pci_renesas idma64 hid video wmi [last unloaded: snd_seq_device]
[ +0,000014] CPU: 4 PID: 894 Comm: btrfs-transacti Tainted: P W O 6.5.0-45-generic #45~22.04.1-Ubuntu
[ +0,000002] Hardware name: Dell Inc. XPS 15 9560
[ +0,000002] RIP: 0010:__btrfs_free_extent+0xd83/0xf10 [btrfs]
[ +0,000061] Code: ff ff 48 c7 c7 c0 e7 1d c4 e8 59 b8 61 be 0f 0b 4c 8b 5d a8 e9 23 fc ff ff be 8b ff ff ff 48 c7 c7 c0 e7 1d c4 e8 3d b8 61 be <0f> 0b e9 3e fc ff ff 8b 7d 80 e8 be ec fe ff 84 c0 0f 85 41 01 00
[ +0,000002] RSP: 0018:ffffafda0052fb70 EFLAGS: 00010246
[ +0,000002] RAX: 0000000000000000 RBX: 0800000084207000 RCX: 0000000000000000
[ +0,000001] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
[ +0,000001] RBP: ffffafda0052fc10 R08: 0000000000000000 R09: 0000000000000000
[ +0,000002] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000001
[ +0,000001] R13: ffff92fa2c3b0d20 R14: ffff92fb936e9c30 R15: ffff92fa407e2770
[ +0,000001] FS: 0000000000000000(0000) GS:ffff92fe7e500000(0000) knlGS:0000000000000000
[ +0,000001] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ +0,000002] CR2: 000000c00053b000 CR3: 00000003b8c94005 CR4: 00000000003706e0
[ +0,000002] Call Trace:
[ +0,000001] <TASK>
[ +0,000001] ? show_regs+0x6d/0x80
[ +0,000003] ? __warn+0x89/0x160
[ +0,000003] ? __btrfs_free_extent+0xd83/0xf10 [btrfs]
[ +0,000061] ? report_bug+0x17e/0x1b0
[ +0,000005] ? handle_bug+0x46/0x90
[ +0,000003] ? exc_invalid_op+0x18/0x80
[ +0,000002] ? asm_exc_invalid_op+0x1b/0x20
[ +0,000004] ? __btrfs_free_extent+0xd83/0xf10 [btrfs]
[ +0,000062] run_delayed_data_ref+0x8b/0x180 [btrfs]
[ +0,000060] btrfs_run_delayed_refs_for_head+0x183/0x550 [btrfs]
[ +0,000060] __btrfs_run_delayed_refs+0xc0/0x150 [btrfs]
[ +0,000065] btrfs_run_delayed_refs+0x6d/0x1f0 [btrfs]
[ +0,000067] btrfs_start_dirty_block_groups+0x36b/0x530 [btrfs]
[ +0,000074] btrfs_commit_transaction+0xb3/0xbb0 [btrfs]
[ +0,000081] ? start_transaction+0xcf/0x660 [btrfs]
[ +0,000113] transaction_kthread+0x14b/0x1c0 [btrfs]
[ +0,000082] ? __pfx_transaction_kthread+0x10/0x10 [btrfs]
[ +0,000065] kthread+0xef/0x120
[ +0,000005] ? __pfx_kthread+0x10/0x10
[ +0,000003] ret_from_fork+0x44/0x70
[ +0,000005] ? __pfx_kthread+0x10/0x10
[ +0,000002] ret_from_fork_asm+0x1b/0x30
[ +0,000004] </TASK>
[ +0,000002] ---[ end trace 0000000000000000 ]---
[ +0,000003] BTRFS: error (device dm-0: state A) in __btrfs_free_extent:3113: errno=-117 Filesystem corrupted
[ +0,000005] BTRFS info (device dm-0: state EA): forced readonly
[ +0,000003] BTRFS info (device dm-0: state EA): leaf 641691598848 gen 755469 total ptrs 91 free space 9475 owner 2
[ +0,000002] item 0 key (723410153472 168 134217728) itemoff 16230 itemsize 53
[ +0,000003] extent refs 1 gen 702195 flags 1
[ +0,000001] ref#0: extent data backref root 256 objectid 27101054 offset 147390464 count 1
[ +0,000003] item 1 key (723544371200 168 134217728) itemoff 16177 itemsize 53
[ +0,000002] extent refs 1 gen 702195 flags 1
[ +0,000001] ref#0: extent data backref root 256 objectid 27101054 offset 281608192 count 1
[ +0,000001] item 2 key (723678588928 168 30998528) itemoff 16124 itemsize 53
[ +0,000002] extent refs 1 gen 702195 flags 1
[ +0,000001] ref#0: extent data backref root 256 objectid 27101054 offset 415825920 count 1
[ +0,000002] item 3 key (723732398080 168 134217728) itemoff 16071 itemsize 53
[ +0,000001] extent refs 1 gen 702198 flags 1
[ +0,000001] ref#0: extent data backref root 256 objectid 27101349 offset 134217728 count 1
[ +0,000002] item 4 key (723732398080 192 1073741824) itemoff 16047 itemsize 24
[ +0,000001] block group used 1061298176 chunk_objectid 256 flags 1
[ +0,000002] item 5 key (723866615808 168 134217728) itemoff 15994 itemsize 53
[ +0,000001] extent refs 1 gen 702198 flags 1
[ +0,000001] ref#0: extent data backref root 256 objectid 27101349 offset 268435456 count 1
[ +0,000002] item 6 key (724000833536 168 134217728) itemoff 15941 itemsize 53
[ +0,000001] extent refs 1 gen 702198 flags 1
[ +0,000001] ref#0: extent data backref root 256 objectid 27101349 offset 536870912 count 1
[ +0,000001] item 7 key (724135051264 168 134217728) itemoff 15888 itemsize 53
[ +0,000002] extent refs 1 gen 702198 flags 1
[ +0,000001] ref#0: extent data backref root 256 objectid 27101350 offset 0 count 1
[ +0,000001] item 8 key (724269268992 168 39886848) itemoff 15835 itemsize 53
[ +0,000002] extent refs 1 gen 702198 flags 1
[ +0,000001] ref#0: extent data backref root 256 objectid 27101350 offset 134217728 count 1
[ +0,000001] item 9 key (724309155840 168 134217728) itemoff 15782 itemsize 53
[ +0,000002] extent refs 1 gen 702200 flags 1
[ +0,000001] ref#0: extent data backref root 256 objectid 27101498 offset 134217728 count 1
[ +0,000001] item 10 key (724443373568 168 134217728) itemoff 15729 itemsize 53
[ +0,000002] extent refs 1 gen 715989 flags 1
[ +0,000001] ref#0: extent data backref root 256 objectid 27240220 offset 939524096 count 1
[ +0,000002] item 11 key (724577591296 168 81887232) itemoff 15676 itemsize 53
[ +0,000001] extent refs 1 gen 715989 flags 1
[ +0,000001] ref#0: extent data backref root 256 objectid 27240220 offset 1073741824 count 1
[ +0,000002] item 12 key (724659478528 168 134217728) itemoff 15623 itemsize 53
[ +0,000001] extent refs 1 gen 717398 flags 1
[ +0,000001] ref#0: extent data backref root 257 objectid 49839032 offset 268435456 count 1
[ +0,000002] item 13 key (724806139904 192 1073741824) itemoff 15599 itemsize 24
[ +0,000002] block group used 913412096 chunk_objectid 256 flags 1
[ +0,000001] item 14 key (724940357632 168 134217728) itemoff 15546 itemsize 53
[ +0,000001] extent refs 1 gen 714517 flags 1
[ +0,000001] ref#0: extent data backref root 257 objectid 49729770 offset 0 count 1
[ +0,000002] item 15 key (725074575360 168 134217728) itemoff 15493 itemsize 53
[ +0,000001] extent refs 1 gen 714517 flags 1
[ +0,000001] ref#0: extent data backref root 257 objectid 49729770 offset 134217728 count 1
[ +0,000002] item 16 key (725208793088 168 134217728) itemoff 15440 itemsize 53
[ +0,000001] extent refs 1 gen 714517 flags 1
[ +0,000001] ref#0: extent data backref root 257 objectid 49729770 offset 268435456 count 1
[ +0,000001] item 17 key (725343010816 168 134217728) itemoff 15387 itemsize 53
[ +0,000002] extent refs 1 gen 714517 flags 1
[ +0,000001] ref#0: extent data backref root 257 objectid 49729770 offset 402653184 count 1
[ +0,000002] item 18 key (725477228544 168 23326720) itemoff 15334 itemsize 53
[ +0,000001] extent refs 1 gen 714517 flags 1
[ +0,000001] ref#0: extent data backref root 257 objectid 49729770 offset 536870912 count 1
[ +0,000002] item 19 key (725500555264 168 134217728) itemoff 15281 itemsize 53
[ +0,000001] extent refs 1 gen 714518 flags 1
[ +0,000001] ref#0: extent data backref root 257 objectid 49729770 offset 560193536 count 1
[ +0,000001] item 20 key (725634772992 168 134217728) itemoff 15228 itemsize 53
[ +0,000002] extent refs 1 gen 714518 flags 1
[ +0,000001] ref#0: extent data backref root 257 objectid 49729770 offset 694411264 count 1
[ +0,000001] item 21 key (725768990720 168 84779008) itemoff 15175 itemsize 53
[ +0,000002] extent refs 1 gen 714518 flags 1
[ +0,000000] ref#0: extent data backref root 257 objectid 49729770 offset 1231282176 count 1
[ +0,000002] item 22 key (725879881728 192 1073741824) itemoff 15151 itemsize 24
[ +0,000002] block group used 776323072 chunk_objectid 256 flags 1
[ +0,000001] item 23 key (726007750656 168 134217728) itemoff 15098 itemsize 53
[ +0,000001] extent refs 1 gen 727840 flags 1
[ +0,000001] ref#0: extent data backref root 256 objectid 27331090 offset 0 count 1
[ +0,000002] item 24 key (726168449024 168 134217728) itemoff 15045 itemsize 53
[ +0,000002] extent refs 1 gen 717398 flags 1
[ +0,000000] ref#0: extent data backref root 257 objectid 49839032 offset 402653184 count 1
[ +0,000003] item 25 key (726352687104 168 43327488) itemoff 14992 itemsize 53
[ +0,000001] extent refs 1 gen 749001 flags 1
[ +0,000001] ref#0: extent data backref root 257 objectid 50936052 offset 134217728 count 1
[ +0,000001] item 26 key (726396014592 168 32387072) itemoff 14939 itemsize 53
[ +0,000002] extent refs 1 gen 749001 flags 1
[ +0,000001] ref#0: extent data backref root 257 objectid 50936052 offset 714412032 count 1
[ +0,000001] item 27 key (726428401664 168 29519872) itemoff 14886 itemsize 53
[ +0,000002] extent refs 1 gen 749002 flags 1
[ +0,000000] ref#0: extent data backref root 257 objectid 50936052 offset 1914474496 count 1
[ +0,000002] item 28 key (726533935104 168 134217728) itemoff 14833 itemsize 53
[ +0,000001] extent refs 1 gen 749001 flags 1
[ +0,000001] ref#0: extent data backref root 257 objectid 50936052 offset 0 count 1
[ +0,000002] item 29 key (726668152832 168 134217728) itemoff 14780 itemsize 53
[ +0,000001] extent refs 1 gen 749001 flags 1
[ +0,000001] ref#0: extent data backref root 257 objectid 50936052 offset 177541120 count 1
[ +0,000002] item 30 key (726802370560 168 134217728) itemoff 14727 itemsize 53
[ +0,000001] extent refs 1 gen 749001 flags 1
[ +0,000001] ref#0: extent data backref root 257 objectid 50936052 offset 311758848 count 1
[ +0,000002] item 31 key (726953623552 168 134217728) itemoff 14674 itemsize 53
[ +0,000001] extent refs 1 gen 720513 flags 1
[ +0,000001] ref#0: extent data backref root 257 objectid 50052713 offset 838959104 count 1
[ +0,000001] item 32 key (726953623552 192 1073741824) itemoff 14650 itemsize 24
[ +0,000002] block group used 1056161792 chunk_objectid 256 flags 1
[ +0,000001] item 33 key (727087841280 168 134217728) itemoff 14597 itemsize 53
[ +0,000001] extent refs 1 gen 720513 flags 1
[ +0,000001] ref#0: extent data backref root 257 objectid 50052713 offset 1020116992 count 1
[ +0,000002] item 34 key (727222059008 168 134217728) itemoff 14544 itemsize 53
[ +0,000001] extent refs 1 gen 720513 flags 1
[ +0,000001] ref#0: extent data backref root 257 objectid 50052713 offset 1154334720 count 1
[ +0,000001] item 35 key (727356276736 168 134217728) itemoff 14491 itemsize 53
[ +0,000002] extent refs 1 gen 720513 flags 1
[ +0,000001] ref#0: extent data backref root 257 objectid 50052713 offset 1288552448 count 1
[ +0,000001] item 36 key (727490494464 168 134217728) itemoff 14438 itemsize 53
[ +0,000002] extent refs 1 gen 720513 flags 1
[ +0,000001] ref#0: extent data backref root 257 objectid 50052713 offset 1422770176 count 1
[ +0,000001] item 37 key (727624712192 168 134217728) itemoff 14385 itemsize 53
[ +0,000002] extent refs 1 gen 720514 flags 1
[ +0,000001] ref#0: extent data backref root 257 objectid 50052713 offset 1556987904 count 1
[ +0,000002] item 38 key (727758929920 168 133251072) itemoff 14332 itemsize 53
[ +0,000002] extent refs 1 gen 720514 flags 1
[ +0,000000] ref#0: extent data backref root 257 objectid 50052713 offset 1691205632 count 1
[ +0,000002] item 39 key (727892180992 168 117604352) itemoff 14279 itemsize 53
[ +0,000002] extent refs 1 gen 720514 flags 1
[ +0,000001] ref#0: extent data backref root 257 objectid 50052713 offset 1824452608 count 1
[ +0,000001] item 40 key (728027365376 168 71106560) itemoff 14226 itemsize 53
[ +0,000002] extent refs 1 gen 727840 flags 1
[ +0,000001] ref#0: extent data backref root 256 objectid 27331090 offset 134217728 count 1
[ +0,000001] item 41 key (728027365376 192 1073741824) itemoff 14202 itemsize 24
[ +0,000002] block group used 1006329856 chunk_objectid 256 flags 1
[ +0,000001] item 42 key (728098471936 168 49319936) itemoff 14149 itemsize 53
[ +0,000001] extent refs 1 gen 741460 flags 1
[ +0,000001] ref#0: extent data backref root 256 objectid 27401619 offset 402653184 count 1
[ +0,000002] item 43 key (728161583104 168 134217728) itemoff 14096 itemsize 53
[ +0,000001] extent refs 1 gen 720512 flags 1
[ +0,000001] ref#0: extent data backref root 257 objectid 50052713 offset 0 count 1
[ +0,000001] item 44 key (728295800832 168 134217728) itemoff 14043 itemsize 53
[ +0,000002] extent refs 1 gen 720512 flags 1
[ +0,000001] ref#0: extent data backref root 257 objectid 50052713 offset 134217728 count 1
[ +0,000001] item 45 key (728430018560 168 33656832) itemoff 13990 itemsize 53
[ +0,000002] extent refs 1 gen 720512 flags 1
[ +0,000001] ref#0: extent data backref root 257 objectid 50052713 offset 268435456 count 1
[ +0,000001] item 46 key (728463675392 168 134217728) itemoff 13937 itemsize 53
[ +0,000002] extent refs 1 gen 720513 flags 1
[ +0,000001] ref#0: extent data backref root 257 objectid 50052713 offset 302088192 count 1
[ +0,000001] item 47 key (728597893120 168 134217728) itemoff 13884 itemsize 53
[ +0,000002] extent refs 1 gen 720513 flags 1
[ +0,000000] ref#0: extent data backref root 257 objectid 50052713 offset 436305920 count 1
[ +0,000002] item 48 key (728732110848 168 134217728) itemoff 13831 itemsize 53
[ +0,000001] extent refs 1 gen 720513 flags 1
[ +0,000001] ref#0: extent data backref root 257 objectid 50052713 offset 570523648 count 1
[ +0,000002] item 49 key (728866328576 168 134217728) itemoff 13778 itemsize 53
[ +0,000001] extent refs 1 gen 720513 flags 1
[ +0,000001] ref#0: extent data backref root 257 objectid 50052713 offset 704741376 count 1
[ +0,000002] item 50 key (729000546304 168 46940160) itemoff 13725 itemsize 53
[ +0,000001] extent refs 1 gen 720513 flags 1
[ +0,000001] ref#0: extent data backref root 257 objectid 50052713 offset 973176832 count 1
[ +0,000002] item 51 key (729101107200 168 134217728) itemoff 13672 itemsize 53
[ +0,000001] extent refs 1 gen 749002 flags 1
[ +0,000001] ref#0: extent data backref root 257 objectid 50936052 offset 881012736 count 1
[ +0,000002] item 52 key (729101107200 192 1073741824) itemoff 13648 itemsize 24
[ +0,000001] block group used 1017196544 chunk_objectid 256 flags 1
[ +0,000001] item 53 key (729235324928 168 134217728) itemoff 13595 itemsize 53
[ +0,000001] extent refs 1 gen 749002 flags 1
[ +0,000001] ref#0: extent data backref root 257 objectid 50936052 offset 1149448192 count 1
[ +0,000002] item 54 key (729373949952 168 134217728) itemoff 13542 itemsize 53
[ +0,000001] extent refs 1 gen 727840 flags 1
[ +0,000001] ref#0: extent data backref root 256 objectid 27331094 offset 0 count 1
[ +0,000001] item 55 key (729508167680 168 77672448) itemoff 13489 itemsize 53
[ +0,000002] extent refs 1 gen 727840 flags 1
[ +0,000001] ref#0: extent data backref root 256 objectid 27331094 offset 134217728 count 1
[ +0,000001] item 56 key (729585840128 168 134217728) itemoff 13436 itemsize 53
[ +0,000002] extent refs 1 gen 749001 flags 1
[ +0,000001] ref#0: extent data backref root 257 objectid 50936052 offset 445976576 count 1
[ +0,000001] item 57 key (729720057856 168 134217728) itemoff 13383 itemsize 53
[ +0,000002] extent refs 1 gen 749001 flags 1
[ +0,000000] ref#0: extent data backref root 257 objectid 50936052 offset 580194304 count 1
[ +0,000002] item 58 key (729854275584 168 134217728) itemoff 13330 itemsize 53
[ +0,000001] extent refs 1 gen 749002 flags 1
[ +0,000001] ref#0: extent data backref root 257 objectid 50936052 offset 746795008 count 1
[ +0,000002] item 59 key (729988493312 168 134217728) itemoff 13277 itemsize 53
[ +0,000001] extent refs 1 gen 749002 flags 1
[ +0,000001] ref#0: extent data backref root 257 objectid 50936052 offset 1015230464 count 1
[ +0,000002] item 60 key (731248590848 168 93347840) itemoff 13224 itemsize 53
[ +0,000001] extent refs 1 gen 749002 flags 1
[ +0,000001] ref#0: extent data backref root 257 objectid 50936052 offset 1417883648 count 1
[ +0,000002] item 61 key (731248590848 192 1073741824) itemoff 13200 itemsize 24
[ +0,000001] block group used 1016864768 chunk_objectid 256 flags 1
[ +0,000001] item 62 key (731373760512 168 134217728) itemoff 13147 itemsize 53
[ +0,000001] extent refs 1 gen 741449 flags 1
[ +0,000001] ref#0: extent data backref root 256 objectid 27395401 offset 268435456 count 1
[ +0,000002] item 63 key (731507978240 168 134217728) itemoff 13094 itemsize 53
[ +0,000001] extent refs 1 gen 741450 flags 1
[ +0,000001] ref#0: extent data backref root 256 objectid 27395418 offset 134217728 count 1
[ +0,000002] item 64 key (731642195968 168 134217728) itemoff 13041 itemsize 53
[ +0,000001] extent refs 1 gen 741450 flags 1
[ +0,000001] ref#0: extent data backref root 256 objectid 27395479 offset 134217728 count 1
[ +0,000002] item 65 key (731776413696 168 26189824) itemoff 12988 itemsize 53
[ +0,000001] extent refs 1 gen 741450 flags 1
[ +0,000001] ref#0: extent data backref root 256 objectid 27395479 offset 268435456 count 1
[ +0,000001] item 66 key (731802603520 168 92020736) itemoff 12935 itemsize 53
[ +0,000002] extent refs 1 gen 741452 flags 1
[ +0,000001] ref#0: extent data backref root 256 objectid 27395583 offset 0 count 1
[ +0,000001] item 67 key (731894624256 168 134217728) itemoff 12882 itemsize 53
[ +0,000002] extent refs 1 gen 741452 flags 1
[ +0,000001] ref#0: extent data backref root 256 objectid 27395632 offset 0 count 1
[ +0,000002] item 68 key (732028841984 168 134217728) itemoff 12829 itemsize 53
[ +0,000001] extent refs 1 gen 741452 flags 1
[ +0,000001] ref#0: extent data backref root 256 objectid 27395633 offset 0 count 1
[ +0,000002] item 69 key (732163059712 168 134217728) itemoff 12776 itemsize 53
[ +0,000001] extent refs 1 gen 741453 flags 1
[ +0,000001] ref#0: extent data backref root 256 objectid 27395655 offset 134217728 count 1
[ +0,000002] item 70 key (732322332672 168 134217728) itemoff 12723 itemsize 53
[ +0,000001] extent refs 1 gen 741453 flags 1
[ +0,000001] ref#0: extent data backref root 256 objectid 27395656 offset 134217728 count 1
[ +0,000002] item 71 key (732322332672 192 1073741824) itemoff 12699 itemsize 24
[ +0,000002] block group used 1032589312 chunk_objectid 256 flags 1
[ +0,000001] item 72 key (732456550400 168 45006848) itemoff 12646 itemsize 53
[ +0,000001] extent refs 1 gen 741453 flags 1
[ +0,000001] ref#0: extent data backref root 256 objectid 27395656 offset 268435456 count 1
[ +0,000002] item 73 key (732501557248 168 134217728) itemoff 12593 itemsize 53
[ +0,000001] extent refs 1 gen 741455 flags 1
[ +0,000001] ref#0: extent data backref root 256 objectid 27395742 offset 0 count 1
[ +0,000002] item 74 key (732635774976 168 10117120) itemoff 12540 itemsize 53
[ +0,000001] extent refs 1 gen 741455 flags 1
[ +0,000001] ref#0: extent data backref root 256 objectid 27395742 offset 134217728 count 1
[ +0,000002] item 75 key (732645892096 168 134217728) itemoff 12487 itemsize 53
[ +0,000001] extent refs 1 gen 741455 flags 1
[ +0,000001] ref#0: extent data backref root 256 objectid 27395764 offset 0 count 1
[ +0,000002] item 76 key (732780109824 168 134217728) itemoff 12434 itemsize 53
[ +0,000001] extent refs 1 gen 741455 flags 1
[ +0,000001] ref#0: extent data backref root 256 objectid 27395764 offset 134217728 count 1
[ +0,000002] item 77 key (732914327552 168 78495744) itemoff 12381 itemsize 53
[ +0,000002] extent refs 1 gen 741455 flags 1
[ +0,000001] ref#0: extent data backref root 256 objectid 27395856 offset 0 count 1
[ +0,000001] item 78 key (732992823296 168 118665216) itemoff 12328 itemsize 53
[ +0,000002] extent refs 1 gen 741455 flags 1
[ +0,000000] ref#0: extent data backref root 256 objectid 27395910 offset 0 count 1
[ +0,000002] item 79 key (733111488512 168 109215744) itemoff 12275 itemsize 53
[ +0,000001] extent refs 1 gen 741460 flags 1
[ +0,000001] ref#0: extent data backref root 256 objectid 27401617 offset 134217728 count 1
[ +0,000002] item 80 key (733220704256 168 134217728) itemoff 12222 itemsize 53
[ +0,000002] extent refs 1 gen 741460 flags 1
[ +0,000001] ref#0: extent data backref root 256 objectid 27401619 offset 134217728 count 1
[ +0,000002] item 81 key (733396074496 192 1073741824) itemoff 12198 itemsize 24
[ +0,000001] block group used 819740672 chunk_objectid 256 flags 1
[ +0,000001] item 82 key (733544726528 168 134217728) itemoff 12145 itemsize 53
[ +0,000002] extent refs 1 gen 745931 flags 1
[ +0,000001] ref#0: extent data backref root 256 objectid 27471460 offset 0 count 1
[ +0,000002] item 83 key (733678944256 168 134217728) itemoff 12092 itemsize 53
[ +0,000001] extent refs 1 gen 745931 flags 1
[ +0,000001] ref#0: extent data backref root 256 objectid 27471460 offset 134217728 count 1
[ +0,000002] item 84 key (733813161984 168 14434304) itemoff 12039 itemsize 53
[ +0,000001] extent refs 1 gen 745931 flags 1
[ +0,000001] ref#0: extent data backref root 256 objectid 27471460 offset 268435456 count 1
[ +0,000002] item 85 key (733827596288 168 134217728) itemoff 11986 itemsize 53
[ +0,000001] extent refs 1 gen 749002 flags 1
[ +0,000001] ref#0: extent data backref root 257 objectid 50936052 offset 1283665920 count 1
[ +0,000002] item 86 key (733961814016 168 134217728) itemoff 11933 itemsize 53
[ +0,000002] extent refs 1 gen 749002 flags 1
[ +0,000001] ref#0: extent data backref root 257 objectid 50936052 offset 1511821312 count 1
[ +0,000002] item 87 key (734096031744 168 134217728) itemoff 11880 itemsize 53
[ +0,000001] extent refs 1 gen 749002 flags 1
[ +0,000001] ref#0: extent data backref root 257 objectid 50936052 offset 1646039040 count 1
[ +0,000002] item 88 key (734230249472 168 134217728) itemoff 11827 itemsize 53
[ +0,000002] extent refs 1 gen 749002 flags 1
[ +0,000001] ref#0: extent data backref root 257 objectid 50936052 offset 1780256768 count 1
[ +0,000001] item 89 key (734469816320 192 1073741824) itemoff 11803 itemsize 24
[ +0,000002] block group used 593920 chunk_objectid 256 flags 1
[ +0,000001] item 90 key (735129747456 168 593920) itemoff 11750 itemsize 53
[ +0,000001] extent refs 1 gen 749002 flags 1
[ +0,000001] ref#0: extent data backref root 257 objectid 50936052 offset 1511231488 count 1
[ +0,000002] BTRFS critical (device dm-0: state EA): unable to find ref byte nr 576460754520141824 parent 0 root 257 owner 50058751 offset 0 slot 90
[ +0,000008] BTRFS error (device dm-0: state EA): failed to run delayed ref for logical 576460754520141824 num_bytes 4096 type 178 action 2 ref_mod 1: -2
[ +0,000005] BTRFS: error (device dm-0: state EA) in btrfs_run_delayed_refs:2187: errno=-2 No such entry
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Errors found in extent allocation tree (single bit flipped)
2024-08-12 14:57 Errors found in extent allocation tree (single bit flipped) Pieter P
@ 2024-08-20 14:01 ` Pieter P
2024-08-20 22:55 ` Qu Wenruo
2024-08-20 23:02 ` Qu Wenruo
1 sibling, 1 reply; 12+ messages in thread
From: Pieter P @ 2024-08-20 14:01 UTC (permalink / raw)
To: linux-btrfs
On 12/08/2024 16:57, Pieter P wrote:
> Can this problem be fixed by running btrfs check --repair from a live
> USB? (I'm using Ubuntu 22.04 with btrfs-progs v5.16.2).
Turns out, the answer is no :(
I tried this on (a copy of) a backup image of the corrupt file system,
and btrfs check --repair simply failed in step [2/7].
1. What should my next move be? How can I fix my file system?
2. Or if that's not possible, how to figure out which files may be
lost/corrupt when copying everything to a fresh btrfs partition?
3. When creating a new file system, can I somehow "clone" the subvolumes
on my current file system (e.g. @ and @home, and the dozens of
subvolumes created by Docker)?
Thanks,
Pieter
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Errors found in extent allocation tree (single bit flipped)
2024-08-20 14:01 ` Pieter P
@ 2024-08-20 22:55 ` Qu Wenruo
0 siblings, 0 replies; 12+ messages in thread
From: Qu Wenruo @ 2024-08-20 22:55 UTC (permalink / raw)
To: Pieter P, linux-btrfs
在 2024/8/20 23:31, Pieter P 写道:
> On 12/08/2024 16:57, Pieter P wrote:
>> Can this problem be fixed by running btrfs check --repair from a live
>> USB? (I'm using Ubuntu 22.04 with btrfs-progs v5.16.2).
>
> Turns out, the answer is no :(
> I tried this on (a copy of) a backup image of the corrupt file system,
> and btrfs check --repair simply failed in step [2/7].
The corruption is not in extent tree, but seemingly inside the subvolume
tree.
And in that case, "btrfs check --mode=lowmem" output may help a little
more, and the same for "btrfs check --repair --mode=lowmem".
>
> 1. What should my next move be? How can I fix my file system?
If above lowmem mode still doesn't work, I can craft a tool for your
specific corruption if really needed.
But that will need the dump of your subvolume 257 inode 50058751.
> 2. Or if that's not possible, how to figure out which files may be
> lost/corrupt when copying everything to a fresh btrfs partition?
Subvolume 257, inode 50058751.
> 3. When creating a new file system, can I somehow "clone" the subvolumes
> on my current file system (e.g. @ and @home, and the dozens of
> subvolumes created by Docker)?
Btrfs send should still be fine, except subvolume 257.
Thanks,
Qu
>
> Thanks,
> Pieter
>
>
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Errors found in extent allocation tree (single bit flipped)
2024-08-12 14:57 Errors found in extent allocation tree (single bit flipped) Pieter P
2024-08-20 14:01 ` Pieter P
@ 2024-08-20 23:02 ` Qu Wenruo
2024-08-26 23:04 ` Pieter P
1 sibling, 1 reply; 12+ messages in thread
From: Qu Wenruo @ 2024-08-20 23:02 UTC (permalink / raw)
To: Pieter P, linux-btrfs
在 2024/8/13 00:27, Pieter P 写道:
> Hello,
>
> My btrfs partition suddenly went read-only three different times over
> the past couple of days. Looking at the dmesg output (see attachment),
> and then running btrfs check, it seems that a single bit is flipped
> somewhere in a btrfs bytenr (2216718336 = 0x84207000, 576460754520141824
> = 0x800000084207000).
>
> I've verified that my RAM, SSD and other hardware are fine using Dell's
> preboot diagnostic tool
Unless the system is using ECC memories, I doubt if that diagnostic tool
makes any difference.
If there is already a strong indication of bitflip (and yes, it is), a
full memtest is highly recommended.
Unless the fs is pretty old and have been run on other systems before.
(aka, the corruption happened on other systems and not detected until
the file is going to be deleted).
Thanks,
Qu
> and Samsung's SSD magician full diagnostic scan.
[...]
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Errors found in extent allocation tree (single bit flipped)
2024-08-20 23:02 ` Qu Wenruo
@ 2024-08-26 23:04 ` Pieter P
2024-08-26 23:18 ` Qu Wenruo
0 siblings, 1 reply; 12+ messages in thread
From: Pieter P @ 2024-08-26 23:04 UTC (permalink / raw)
To: Qu Wenruo, linux-btrfs
[-- Attachment #1: Type: text/plain, Size: 1590 bytes --]
Thanks for the help!
On 21/08/2024 00:55, Qu Wenruo wrote:
> And in that case, "btrfs check --mode=lowmem" output may help a little
> more, and the same for "btrfs check --repair --mode=lowmem".
I've attached the output from both commands, unfortunately, that didn't
fix the issue.
Subvolume 257 is my /home directory, and the broken inode is a temporary
file in a Chromium cache folder. I've tried deleting and overwriting
that file, but this causes the file system to go read-only a couple of
seconds later, and after re-mounting, the file re-appears. Reading the
file is not possible (input/output error).
Is there a way to restore the file system by simply deleting this
inode+data entirely?
> If above lowmem mode still doesn't work, I can craft a tool for your
> specific corruption if really needed.
>
> But that will need the dump of your subvolume 257 inode 50058751.
Which data would you need specifically? How do I get such a dump?
> Unless the system is using ECC memories, I doubt if that diagnostic tool
> makes any difference.
>
> If there is already a strong indication of bitflip (and yes, it is), a
> full memtest is highly recommended.
No ECC memory, but I did perform the "thorough" memory test using the
Dell diagnostic tool multiple times. It included data bus stress tests,
march C- tests, ground bounce tests, random pattern tests, and some
others; all passed without issues. Since I haven't noticed any other
problems (the system works fine when booting from another drive), I'm
blaming an unfortunate cosmic ray for now :)
Thank you!
Pieter
[-- Attachment #2: btrfs-output.txt --]
[-- Type: text/plain, Size: 3551 bytes --]
sudo ./btrfs.static check --mode=lowmem /dev/mapper/luks-xxxxx
Opening filesystem to check...
Checking filesystem on /dev/mapper/luks-xxxxx
UUID: xxxxx
[1/7] checking root items
[2/7] checking extents
ERROR: extent[2216718336, 4096] referencer count mismatch (root: 257, owner: 50058751, offset: 0) wanted: 1, have: 0
ERROR: file extent[50058751 0] root 257 owner 257 backref lost
ERROR: errors found in extent allocation tree or chunk allocation
[3/7] checking free space tree
[4/7] checking fs roots
ERROR: root 257 EXTENT_DATA[50058751 0] csum missing, have: 0, expected: 4096
ERROR: root 257 EXTENT_DATA[50058751 0] compressed extent must have csum, but only 0 bytes have, expect 4096
ERROR: errors found in fs roots
found 597039108096 bytes used, error(s) found
total csum bytes: 563978828
total tree bytes: 14739062784
total fs tree bytes: 13088227328
total extent tree bytes: 911917056
btree space waste bytes: 2527601892
file data blocks allocated: 816812535808
referenced 1055290490880
sudo ./btrfs.static check --mode=lowmem --repair /dev/mapper/luks-xxxxx
enabling repair mode
WARNING:
Do not use --repair unless you are advised to do so by a developer
or an experienced user, and then only after having accepted that no
fsck can successfully repair all types of filesystem corruption. E.g.
some software or hardware bugs can fatally damage a volume.
The operation will start in 10 seconds.
Use Ctrl-C to stop it.
10 9 8 7 6 5 4 3 2 1
Starting repair.
Opening filesystem to check...
Checking filesystem on /dev/mapper/luks-xxxxx
UUID: xxxxx
[1/7] checking root items
Fixed 0 roots.
[2/7] checking extents
ERROR: extent[2216718336, 4096] referencer count mismatch (root: 257, owner: 50058751, offset: 0) wanted: 1, have: 0
Created new chunk [741986009088 1073741824]
Delete backref in extent [2216718336 4096]
ERROR: file extent[50058751 0] root 257 owner 257 backref lost
Add one extent data backref [576460754520141824 4096]
super bytes_used 597040041984 mismatches actual used 597040926720
ERROR: errors found in extent allocation tree or chunk allocation
[3/7] checking free space tree
[4/7] checking fs roots
ERROR: root 257 EXTENT_DATA[50058751 0] csum missing, have: 0, expected: 4096
ERROR: root 257 EXTENT_DATA[50058751 0] compressed extent must have csum, but only 0 bytes have, expect 4096
ERROR: errors found in fs roots
found 597040926720 bytes used, error(s) found
total csum bytes: 563978824
total tree bytes: 14740029440
total fs tree bytes: 13088227328
total extent tree bytes: 911949824
btree space waste bytes: 2527686993
file data blocks allocated: 816812535808
referenced 1055290490880
sudo ./btrfs.static check /dev/mapper/luks-xxxxx
Opening filesystem to check...
Checking filesystem on /dev/mapper/luks-xxxxx
UUID: xxxxx
[1/7] checking root items
[2/7] checking extents
super bytes used 1194081968128 mismatches actual used 597041922048
ERROR: errors found in extent allocation tree or chunk allocation
[3/7] checking free space tree
[4/7] checking fs roots
root 257 inode 50058751 errors 1000, some csum missing
ERROR: errors found in fs roots
found 597033414662 bytes used, error(s) found
total csum bytes: 563978824
total tree bytes: 14741880832
total fs tree bytes: 13088227328
total extent tree bytes: 911933440
btree space waste bytes: 2527859565
file data blocks allocated: 816812535808
referenced 1055290490880
sudo btrfs inspect-internal inode-resolve 50058751 /media/pieter/xxxxx/@home
/media/pieter/xxxxx/@home/pieter/snap/chromium/common/chromium/Default/Cache/Cache_Data/f88d9d46e1ab64e3_0
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Errors found in extent allocation tree (single bit flipped)
2024-08-26 23:04 ` Pieter P
@ 2024-08-26 23:18 ` Qu Wenruo
2024-08-27 0:47 ` Pieter P
0 siblings, 1 reply; 12+ messages in thread
From: Qu Wenruo @ 2024-08-26 23:18 UTC (permalink / raw)
To: Pieter P, linux-btrfs
在 2024/8/27 08:34, Pieter P 写道:
> Thanks for the help!
>
> On 21/08/2024 00:55, Qu Wenruo wrote:
>
>> And in that case, "btrfs check --mode=lowmem" output may help a little
>> more, and the same for "btrfs check --repair --mode=lowmem".
>
> I've attached the output from both commands, unfortunately, that didn't
> fix the issue.
>
> Subvolume 257 is my /home directory, and the broken inode is a temporary
> file in a Chromium cache folder. I've tried deleting and overwriting
> that file, but this causes the file system to go read-only a couple of
> seconds later, and after re-mounting, the file re-appears. Reading the
> file is not possible (input/output error).
>
> Is there a way to restore the file system by simply deleting this
> inode+data entirely?
After your latest --repair try, the fs should be more or less fine, you
can try remove the offending file.
Just a minor problem left with the superblock used bytes, that can be
fixed by another "btrfs check --repair" run very safely.
>
>> If above lowmem mode still doesn't work, I can craft a tool for your
>> specific corruption if really needed.
>>
>> But that will need the dump of your subvolume 257 inode 50058751.
>
> Which data would you need specifically? How do I get such a dump?
No need anymore, the latest result contains the all the info I need.
>
>> Unless the system is using ECC memories, I doubt if that diagnostic tool
>> makes any difference.
>>
>> If there is already a strong indication of bitflip (and yes, it is), a
>> full memtest is highly recommended.
>
> No ECC memory, but I did perform the "thorough" memory test using the
> Dell diagnostic tool multiple times. It included data bus stress tests,
> march C- tests, ground bounce tests, random pattern tests, and some
> others; all passed without issues. Since I haven't noticed any other
> problems (the system works fine when booting from another drive), I'm
> blaming an unfortunate cosmic ray for now :)
Unfortunately it's indeed a bitflip, from your latest lowmem repair
result (which mostly solves the problem).
> ERROR: extent[2216718336, 4096] referencer count mismatch (root: 257,
owner: 50058751, offset: 0) wanted: 1, have: 0
This is the original extent item, which looks sane to me.
> Add one extent data backref [576460754520141824 4096]
This is the one to be added according to the subvoluem tree. The huge
value itself is already a little concerning.
Furthermore:
hex(576460754520141824) = 0x800000084207000
hex(2216718336) = 0x000000084207000
It's an obvious bitflip.
Another possibility is, the fs is old and you just migrated the drive/fs
to the current platform, but I doubt about the case considering the file
is some browser cache, which shouldn't be that old.
Just in case, mind to try something like memtest86+ (UEFI payload) or
memtester (inside the OS) to rule out the hardware problem?
Hope it's really a cosmic ray, not a persistent hardware problem.
Thanks,
Qu
>
> Thank you!
> Pieter
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Errors found in extent allocation tree (single bit flipped)
2024-08-26 23:18 ` Qu Wenruo
@ 2024-08-27 0:47 ` Pieter P
2024-08-27 5:19 ` Qu Wenruo
0 siblings, 1 reply; 12+ messages in thread
From: Pieter P @ 2024-08-27 0:47 UTC (permalink / raw)
To: Qu Wenruo, linux-btrfs
[-- Attachment #1: Type: text/plain, Size: 979 bytes --]
On 27/08/2024 01:18, Qu Wenruo wrote:
> After your latest --repair try, the fs should be more or less fine, you
> can try remove the offending file.
>
> Just a minor problem left with the superblock used bytes, that can be
> fixed by another "btrfs check --repair" run very safely.
No luck, unfortunately. Removing the file fails (automatically gets
re-mounted read-only, and the file re-appears), and btrfs check --repair
fails to repair the super bytes_used error (output attached).
> Another possibility is, the fs is old and you just migrated the drive/fs
> to the current platform, but I doubt about the case considering the file
> is some browser cache, which shouldn't be that old.
>
> Just in case, mind to try something like memtest86+ (UEFI payload) or
> memtester (inside the OS) to rule out the hardware problem?
The fs is only around 1.5 years old, it never left the system. The file
was last modified on July 22. I'll run another memtest.
Thank you,
Pieter
[-- Attachment #2: btrfs-output-2.txt --]
[-- Type: text/plain, Size: 3102 bytes --]
./btrfs.static --version
btrfs-progs v6.10.1
-EXPERIMENTAL -INJECT +STATIC +LZO +ZSTD +UDEV +FSVERITY +ZONED CRYPTO=builtin
sudo ./btrfs.static check --repair --mode=lowmem /dev/mapper/luks-xxxxx
enabling repair mode
[...]
Starting repair.
Opening filesystem to check...
Checking filesystem on /dev/mapper/luks-xxxxx
UUID: xxxxx
[1/7] checking root items
Fixed 0 roots.
[2/7] checking extents
ERROR: extent[2216718336, 4096] referencer count mismatch (root: 257, owner: 50058751, offset: 0) wanted: 1, have: 0
Created new chunk [741986009088 1073741824]
Delete backref in extent [2216718336 4096]
ERROR: file extent[50058751 0] root 257 owner 257 backref lost
Add one extent data backref [576460754520141824 4096]
super bytes_used 596757958656 mismatches actual used 596757991424
ERROR: errors found in extent allocation tree or chunk allocation
[3/7] checking free space tree
[4/7] checking fs roots
ERROR: root 257 EXTENT_DATA[50058751 0] csum missing, have: 0, expected: 4096
ERROR: root 257 EXTENT_DATA[50058751 0] compressed extent must have csum, but only 0 bytes have, expect 4096
ERROR: errors found in fs roots
found 596757991424 bytes used, error(s) found
total csum bytes: 563713420
total tree bytes: 14721335296
total fs tree bytes: 13072826368
total extent tree bytes: 908722176
btree space waste bytes: 2519622868
file data blocks allocated: 817170100224
referenced 1054998925312
# mount
rm /media/pieter/xxxxx/@home/pieter/snap/chromium/common/chromium/Default/Cache/Cache_Data/f88d9d46e1ab64e3_0
echo $?
0
ls /media/pieter/xxxxx/@home/pieter/snap/chromium/common/chromium/Default/Cache/Cache_Data/f88d9d46e1ab64e3_0
ls: cannot access '/media/pieter/xxxxx/@home/pieter/snap/chromium/common/chromium/Default/Cache/Cache_Data/f88d9d46e1ab64e3_0': No such file or directory
# unmount
sudo ./btrfs.static check --repair /dev/mapper/luks-xxxxx
enabling repair mode
[...]
Starting repair.
Opening filesystem to check...
Checking filesystem on /dev/mapper/luks-xxxxx
UUID: xxxxx
[1/7] checking root items
Fixed 0 roots.
[2/7] checking extents
failed to repair damaged filesystem, aborting
sudo ./btrfs.static check /dev/mapper/luks-xxxxx
Opening filesystem to check...
Checking filesystem on /dev/mapper/luks-xxxxx
UUID: xxxxx
[1/7] checking root items
[2/7] checking extents
super bytes used 1193518342144 mismatches actual used 596760379392
ERROR: errors found in extent allocation tree or chunk allocation
[3/7] checking free space tree
[4/7] checking fs roots
root 257 inode 50058751 errors 1000, some csum missing
ERROR: errors found in fs roots
found 596752400391 bytes used, error(s) found
total csum bytes: 563713420
total tree bytes: 14723727360
total fs tree bytes: 13072826368
total extent tree bytes: 908689408
btree space waste bytes: 2519842792
file data blocks allocated: 817170100224
referenced 1054998925312
# mount
ls /media/pieter/xxxxx/@home/pieter/snap/chromium/common/chromium/Default/Cache/Cache_Data/f88d9d46e1ab64e3_0
/media/pieter/xxxxx/@home/pieter/snap/chromium/common/chromium/Default/Cache/Cache_Data/f88d9d46e1ab64e3_0
# file was not actually removed
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Errors found in extent allocation tree (single bit flipped)
2024-08-27 0:47 ` Pieter P
@ 2024-08-27 5:19 ` Qu Wenruo
2024-08-27 6:51 ` Qu Wenruo
0 siblings, 1 reply; 12+ messages in thread
From: Qu Wenruo @ 2024-08-27 5:19 UTC (permalink / raw)
To: Pieter P, linux-btrfs
在 2024/8/27 10:17, Pieter P 写道:
> On 27/08/2024 01:18, Qu Wenruo wrote:
>> After your latest --repair try, the fs should be more or less fine, you
>> can try remove the offending file.
>>
>> Just a minor problem left with the superblock used bytes, that can be
>> fixed by another "btrfs check --repair" run very safely.
>
> No luck, unfortunately. Removing the file fails (automatically gets
> re-mounted read-only, and the file re-appears), and btrfs check --repair
> fails to repair the super bytes_used error (output attached).
>
>> Another possibility is, the fs is old and you just migrated the drive/fs
>> to the current platform, but I doubt about the case considering the file
>> is some browser cache, which shouldn't be that old.
>>
>> Just in case, mind to try something like memtest86+ (UEFI payload) or
>> memtester (inside the OS) to rule out the hardware problem?
>
> The fs is only around 1.5 years old, it never left the system. The file
> was last modified on July 22. I'll run another memtest.
Anyway considering it's not going to be resolved by repair, I can craft
a dedicated tool, if you can compile btrfs-progs.
That should fix modify that specific file extent (root 257 ino 50058751
filepos 0) by changing its on-disk bytenr to the correctly.
Will let you know when the tool is done (should be pretty soon)
Thanks,
Qu
>
> Thank you,
> Pieter
>
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Errors found in extent allocation tree (single bit flipped)
2024-08-27 5:19 ` Qu Wenruo
@ 2024-08-27 6:51 ` Qu Wenruo
2024-08-27 9:33 ` Qu Wenruo
0 siblings, 1 reply; 12+ messages in thread
From: Qu Wenruo @ 2024-08-27 6:51 UTC (permalink / raw)
To: Pieter P, linux-btrfs
在 2024/8/27 14:49, Qu Wenruo 写道:
>
>
> 在 2024/8/27 10:17, Pieter P 写道:
>> On 27/08/2024 01:18, Qu Wenruo wrote:
>>> After your latest --repair try, the fs should be more or less fine, you
>>> can try remove the offending file.
>>>
>>> Just a minor problem left with the superblock used bytes, that can be
>>> fixed by another "btrfs check --repair" run very safely.
>>
>> No luck, unfortunately. Removing the file fails (automatically gets
>> re-mounted read-only, and the file re-appears), and btrfs check --repair
>> fails to repair the super bytes_used error (output attached).
>>
>>> Another possibility is, the fs is old and you just migrated the drive/fs
>>> to the current platform, but I doubt about the case considering the file
>>> is some browser cache, which shouldn't be that old.
>>>
>>> Just in case, mind to try something like memtest86+ (UEFI payload) or
>>> memtester (inside the OS) to rule out the hardware problem?
>>
>> The fs is only around 1.5 years old, it never left the system. The file
>> was last modified on July 22. I'll run another memtest.
>
> Anyway considering it's not going to be resolved by repair, I can craft
> a dedicated tool, if you can compile btrfs-progs.
>
> That should fix modify that specific file extent (root 257 ino 50058751
> filepos 0) by changing its on-disk bytenr to the correctly.
>
> Will let you know when the tool is done (should be pretty soon)
Here is the hand crafted fix for you:
https://github.com/adam900710/btrfs-progs/tree/dirty_fixes
The usage is:
# ./btrfs-corrupt-block -X <device>
The fs must be unmounted.
Thanks,
Qu
>
> Thanks,
> Qu
>>
>> Thank you,
>> Pieter
>>
>
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Errors found in extent allocation tree (single bit flipped)
2024-08-27 6:51 ` Qu Wenruo
@ 2024-08-27 9:33 ` Qu Wenruo
2024-08-28 8:49 ` Pieter P
0 siblings, 1 reply; 12+ messages in thread
From: Qu Wenruo @ 2024-08-27 9:33 UTC (permalink / raw)
To: Pieter P, linux-btrfs
在 2024/8/27 16:21, Qu Wenruo 写道:
>
>
> 在 2024/8/27 14:49, Qu Wenruo 写道:
>>
>>
>> 在 2024/8/27 10:17, Pieter P 写道:
>>> On 27/08/2024 01:18, Qu Wenruo wrote:
>>>> After your latest --repair try, the fs should be more or less fine, you
>>>> can try remove the offending file.
>>>>
>>>> Just a minor problem left with the superblock used bytes, that can be
>>>> fixed by another "btrfs check --repair" run very safely.
>>>
>>> No luck, unfortunately. Removing the file fails (automatically gets
>>> re-mounted read-only, and the file re-appears), and btrfs check --repair
>>> fails to repair the super bytes_used error (output attached).
>>>
>>>> Another possibility is, the fs is old and you just migrated the
>>>> drive/fs
>>>> to the current platform, but I doubt about the case considering the
>>>> file
>>>> is some browser cache, which shouldn't be that old.
>>>>
>>>> Just in case, mind to try something like memtest86+ (UEFI payload) or
>>>> memtester (inside the OS) to rule out the hardware problem?
>>>
>>> The fs is only around 1.5 years old, it never left the system. The file
>>> was last modified on July 22. I'll run another memtest.
>>
>> Anyway considering it's not going to be resolved by repair, I can craft
>> a dedicated tool, if you can compile btrfs-progs.
>>
>> That should fix modify that specific file extent (root 257 ino 50058751
>> filepos 0) by changing its on-disk bytenr to the correctly.
>>
>> Will let you know when the tool is done (should be pretty soon)
>
> Here is the hand crafted fix for you:
>
> https://github.com/adam900710/btrfs-progs/tree/dirty_fixes
>
> The usage is:
>
> # ./btrfs-corrupt-block -X <device>
>
> The fs must be unmounted.
And it's better to run "btrfs check --readonly" after a successful run
(the successful should result no output), and paste the output of the
btrfs-check.
If there is something wrong, the program should output something and
please paste it here.
Thanks,
Qu
>
> Thanks,
> Qu
>
>>
>> Thanks,
>> Qu
>>>
>>> Thank you,
>>> Pieter
>>>
>>
>
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Errors found in extent allocation tree (single bit flipped)
2024-08-27 9:33 ` Qu Wenruo
@ 2024-08-28 8:49 ` Pieter P
2024-08-28 9:06 ` Qu Wenruo
0 siblings, 1 reply; 12+ messages in thread
From: Pieter P @ 2024-08-28 8:49 UTC (permalink / raw)
To: Qu Wenruo, linux-btrfs
[-- Attachment #1: Type: text/plain, Size: 858 bytes --]
Thanks, I really appreciate the help!
On 27/08/2024 11:33, Qu Wenruo wrote:
> And it's better to run "btrfs check --readonly" after a successful run
> (the successful should result no output), and paste the output of the
> btrfs-check.
>
> If there is something wrong, the program should output something and
> please paste it here.
btrfs_search_slot did not seem to like the combination of trans=NULL and
cow=1, so I tried both btrfs_search_slot(NULL, root, &key, &path, 0, 0)
and btrfs_search_slot(trans, root, &key, &path, 0, 1), and the latter
fixed the issue (see attachment).
I no longer get any errors from btrfs check, and I've verified the
contents of the file system against an older backup, all important data
is still intact, and everything seems to be okay now.
Thank you very much for taking the time to help resolve this issue!
Pieter
[-- Attachment #2: btrfs-output-3.txt --]
[-- Type: text/plain, Size: 2986 bytes --]
sudo gdb --args ./btrfs-corrupt-block -X /dev/mapper/luks-xxxxx
Program received signal SIGSEGV, Segmentation fault.
0x000055555556d251 in btrfs_cow_block (trans=trans@entry=0x0, root=root@entry=0x555556120920, buf=0x555556124c80, parent=0x0, parent_slot=0, cow_ret=cow_ret@entry=0x7fffffffdf78, nest=BTRFS_NESTING_NORMAL) at kernel-shared/ctree.c:675
675 if (trans->transid != root->fs_info->generation) {
(gdb) bt
#0 0x000055555556d251 in btrfs_cow_block (trans=trans@entry=0x0, root=root@entry=0x555556120920, buf=0x555556124c80, parent=0x0, parent_slot=0, cow_ret=cow_ret@entry=0x7fffffffdf78, nest=BTRFS_NESTING_NORMAL)
at kernel-shared/ctree.c:675
#1 0x00005555555702e3 in btrfs_search_slot (trans=trans@entry=0x0, root=root@entry=0x555556120920, key=key@entry=0x7fffffffe180, p=p@entry=0x7fffffffe1b0, ins_len=ins_len@entry=0, cow=cow@entry=1)
at kernel-shared/ctree.c:1324
#2 0x0000555555563fd3 in dirty_fix (fs_info=<optimized out>) at btrfs-corrupt-block.c:1335
#3 main (argc=<optimized out>, argv=<optimized out>) at btrfs-corrupt-block.c:1528
# Edit btrfs-corrupt-block.c
1335: ret = btrfs_search_slot(NULL, root, &key, &path, 0, 0);
make
sudo gdb --args ./btrfs-corrupt-block -X /dev/mapper/luks-xxxxx
echo $?
0
sudo ../btrfs.static check --readonly /dev/mapper/luks-xxxxx
Opening filesystem to check...
Checking filesystem on /dev/mapper/luks-xxxxx
UUID: xxxxx
[1/7] checking root items
[2/7] checking extents
ref mismatch on [2216718336 4096] extent item 1, found 0
data extent[2216718336, 4096] bytenr mimsmatch, extent item bytenr 2216718336 file item bytenr 0
data extent[2216718336, 4096] referencer count mismatch (root 257 owner 50058751 offset 0) wanted 1 have 0
backpointer mismatch on [2216718336 4096]
owner ref check failed [2216718336 4096]
ref mismatch on [576460754520141824 4096] extent item 0, found 1
data extent[576460754520141824, 4096] referencer count mismatch (root 257 owner 50058751 offset 0) wanted 0 have 1
backpointer mismatch on [576460754520141824 4096]
ERROR: errors found in extent allocation tree or chunk allocation
[3/7] checking free space tree
[4/7] checking fs roots
^C
# Edit btrfs-corrupt-block.c
1335: ret = btrfs_search_slot(trans, root, &key, &path, 0, 1);
make
sudo gdb --args ./btrfs-corrupt-block -X /dev/mapper/luks-xxxxx
echo $?
0
sudo ../btrfs.static check --readonly /dev/mapper/luks-xxxxx
Opening filesystem to check...
Checking filesystem on /dev/mapper/luks-xxxxx
UUID: xxxxx
[1/7] checking root items
[2/7] checking extents
[3/7] checking free space tree
[4/7] checking fs roots
[5/7] checking only csums items (without verifying data)
[6/7] checking root refs
[7/7] checking quota groups skipped (not enabled on this FS)
found 597031350277 bytes used, no error found
total csum bytes: 563978828
total tree bytes: 14739800064
total fs tree bytes: 13088227328
total extent tree bytes: 911933440
btree space waste bytes: 2527709036
file data blocks allocated: 816812535808
referenced 1055290490880
^ permalink raw reply [flat|nested] 12+ messages in thread* Re: Errors found in extent allocation tree (single bit flipped)
2024-08-28 8:49 ` Pieter P
@ 2024-08-28 9:06 ` Qu Wenruo
0 siblings, 0 replies; 12+ messages in thread
From: Qu Wenruo @ 2024-08-28 9:06 UTC (permalink / raw)
To: Pieter P, linux-btrfs
在 2024/8/28 18:19, Pieter P 写道:
> Thanks, I really appreciate the help!
>
> On 27/08/2024 11:33, Qu Wenruo wrote:
>> And it's better to run "btrfs check --readonly" after a successful run
>> (the successful should result no output), and paste the output of the
>> btrfs-check.
>>
>> If there is something wrong, the program should output something and
>> please paste it here.
> btrfs_search_slot did not seem to like the combination of trans=NULL and
> cow=1, so I tried both btrfs_search_slot(NULL, root, &key, &path, 0, 0)
> and btrfs_search_slot(trans, root, &key, &path, 0, 1), and the latter
> fixed the issue (see attachment).
My bad, forgot to pass trans handler.
Thanks a lot for fixing my mistake.
>
> I no longer get any errors from btrfs check, and I've verified the
> contents of the file system against an older backup, all important data
> is still intact, and everything seems to be okay now.
>
> Thank you very much for taking the time to help resolve this issue!
> Pieter
No problem, great that your fs got a full recovery.
Thanks,
Qu
^ permalink raw reply [flat|nested] 12+ messages in thread
end of thread, other threads:[~2024-08-28 9:06 UTC | newest]
Thread overview: 12+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2024-08-12 14:57 Errors found in extent allocation tree (single bit flipped) Pieter P
2024-08-20 14:01 ` Pieter P
2024-08-20 22:55 ` Qu Wenruo
2024-08-20 23:02 ` Qu Wenruo
2024-08-26 23:04 ` Pieter P
2024-08-26 23:18 ` Qu Wenruo
2024-08-27 0:47 ` Pieter P
2024-08-27 5:19 ` Qu Wenruo
2024-08-27 6:51 ` Qu Wenruo
2024-08-27 9:33 ` Qu Wenruo
2024-08-28 8:49 ` Pieter P
2024-08-28 9:06 ` Qu Wenruo
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox