* BTRFS error (device dm-4): failed to run delayed ref for logical 350223581184 num_bytes 16384 type 176 action 1 ref_mod 1: -117 (kernel 6.11.2)
@ 2025-02-06 17:07 Marc MERLIN
2025-02-06 17:34 ` Marc MERLIN
` (2 more replies)
0 siblings, 3 replies; 17+ messages in thread
From: Marc MERLIN @ 2025-02-06 17:07 UTC (permalink / raw)
To: linux-btrfs
Balance ended with btrfs_run_delayed_refs:2199: errno=-117 Filesystem corrupted
btrfs check says it's not
sauron:~# btrfs check /dev/mapper/pool1
Opening filesystem to check...
Checking filesystem on /dev/mapper/pool1
UUID: 4542883b-d8bc-4d7f-8a2e-944dc355dc44
[1/7] checking root items
[2/7] checking extents
[3/7] checking free space tree
[4/7] checking fs roots
[5/7] checking only csums items (without verifying data)
[6/7] checking root refs
[7/7] checking quota groups skipped (not enabled on this FS)
found 228820946944 bytes used, no error found
total csum bytes: 219270232
total tree bytes: 4539334656
total fs tree bytes: 3719593984
total extent tree bytes: 481935360
btree space waste bytes: 1075469196
file data blocks allocated: 15390875832320
referenced 290833076224
Any ideas? this obviously caused downtime, but after the btrfs check saying I'm supposedly
ok, I'm back up for now, hoping it won't happen again and hope is not a strategy :)
Is it safe to run balance again? any idea why it failed?
[1821331.015652] BTRFS info (device dm-4): balance: start -dusage=20
[1821331.015805] BTRFS info (device dm-4): relocating block group 761212698624 flags data
[1821331.090338] BTRFS info (device dm-4): found 31 extents, stage: move data extents
[1821331.237707] BTRFS info (device dm-4): leaf 471333519360 gen 4808182 total ptrs 168 free space 3533 owner 2
[1821331.237716] item 0 key (350222417920 169 0) itemoff 16250 itemsize 33
[1821331.237718] extent refs 1 gen 2907391 flags 2
[1821331.237719] ref#0: tree block backref root 398
(...(
[1821331.238559] item 167 key (350225678336 169 0) itemoff 7733 itemsize 168
[1821331.238560] extent refs 16 gen 4619087 flags 2
[1821331.238561] ref#0: tree block backref root 398
[1821331.238562] ref#1: shared block backref parent 737084112896
[1821331.238563] ref#2: shared block backref parent 736609173504
[1821331.238564] ref#3: shared block backref parent 471099588608
[1821331.238565] ref#4: shared block backref parent 471017488384
[1821331.238567] ref#5: shared block backref parent 470665625600
[1821331.238568] ref#6: shared block backref parent 350806835200
[1821331.238569] ref#7: shared block backref parent 350292066304
[1821331.238570] ref#8: shared block backref parent 349856350208
[1821331.238571] ref#9: shared block backref parent 153429573632
[1821331.238572] ref#10: shared block backref parent 153014337536
[1821331.238573] ref#11: shared block backref parent 152976048128
[1821331.238575] ref#12: shared block backref parent 152753946624
[1821331.238576] ref#13: shared block backref parent 152639225856
[1821331.238577] ref#14: shared block backref parent 50782617600
[1821331.238578] ref#15: shared block backref parent 394002432
[1821331.238579] BTRFS critical (device dm-4): adding refs to an existing tree ref, bytenr 350223581184 num_bytes 16384 root_objectid 398 slot 51
[1821331.238582] BTRFS error (device dm-4): failed to run delayed ref for logical 350223581184 num_bytes 16384 type 176 action 1 ref_mod 1: -117
[1821331.238584] ------------[ cut here ]------------
[1821331.238585] BTRFS: Transaction aborted (error -117)
[1821331.238593] WARNING: CPU: 1 PID: 2457672 at fs/btrfs/extent-tree.c:2199 btrfs_run_delayed_refs+0x107/0x140
[1821331.238599] Modules linked in: mmc_block exfat uinput rpcsec_gss_krb5 nfsv4 dns_resolver nfs netfs sg uas usb_storage nf_conntrack_netlink xfrm_user xfrm_algo xt_addrtype br_netfilter bridge stp llc xt_tcpudp xt_conntrack rfcomm snd_seq_dummy snd_hrtimer ccm overlay ipt_REJECT nf_reject_ipv4 xt_MASQUERADE xt_LOG nf_log_syslog nft_compat nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 cmac algif_hash algif_skcipher af_alg nf_tables bnep binfmt_misc uvcvideo videobuf2_vmalloc uvc videobuf2_memops btusb videobuf2_v4l2 btrtl btbcm videodev btmtk btintel videobuf2_common mc bluetooth nls_utf8 nls_cp437 vfat fat squashfs loop snd_hda_codec_hdmi iwlmvm snd_hda_codec_realtek snd_hda_codec_generic snd_soc_dmic snd_hda_scodec_component mac80211 intel_uncore_frequency snd_sof_pci_intel_tgl intel_uncore_frequency_common snd_sof_pci_intel_cnl intel_tcc_cooling snd_sof_intel_hda_generic snd_sof_intel_hda_common x86_pkg_temp_thermal snd_sof_intel_hda intel_powerclamp libarc4 snd_sof_pci snd_sof_xtensa_dsp kvm_intel
[1821331.238644] snd_soc_hdac_hda iwlwifi kvm thinkpad_acpi snd_soc_acpi_intel_match mei_hdcp mei_pxp snd_soc_acpi cfg80211 processor_thermal_device_pci_legacy nvram tpm_crb processor_thermal_device rapl platform_profile processor_thermal_wt_hint processor_thermal_rfim snd_soc_avs processor_thermal_rapl ucsi_acpi intel_rapl_common nvidiafb processor_thermal_wt_req intel_cstate think_lmi vgastate iTCO_wdt typec_ucsi snd_soc_hda_codec pcspkr processor_thermal_power_floor firmware_attributes_class wmi_bmof snd_hda_intel ee1004 iTCO_vendor_support fb_ddc typec processor_thermal_mbox mei_me roles intel_soc_dts_iosf int3403_thermal rfkill int340x_thermal_zone ac intel_pmc_core intel_vsec tpm_tis tpm_tis_core int3400_thermal pmt_telemetry acpi_pad intel_hid acpi_thermal_rel sparse_keymap pmt_class acpi_tad input_leds evdev joydev serio_raw vboxdrv(OE) soundwire_intel soundwire_cadence snd_sof_intel_hda_mlink soundwire_generic_allocation snd_sof_probes snd_sof snd_sof_utils snd_intel_dspcfg snd_intel_sdw_acpi snd_soc_skl_hda_dsp
[1821331.238682] snd_soc_intel_hda_dsp_common snd_hda_codec snd_hwdep snd_soc_hdac_hdmi snd_hda_ext_core snd_soc_core snd_compress snd_pcm_dmaengine snd_hda_core snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_midi snd_seq_midi_event snd_seq snd_timer snd_rawmidi snd_seq_device snd_ctl_led snd soundcore ac97_bus configs coretemp msr fuse efi_pstore nfsd auth_rpcgss nfs_acl lockd grace sunrpc nfnetlink ip_tables x_tables autofs4 essiv authenc dm_crypt trusted asn1_encoder tee tpm rng_core libaescfb ecdh_generic dm_mod ecc raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx sata_sil24 r8169 realtek mdio_devres libphy mii hid_generic usbhid hid i915 crct10dif_pclmul drm_buddy i2c_algo_bit rtsx_pci_sdmmc crc32_pclmul xhci_pci ttm mmc_core crc32c_intel xhci_hcd polyval_clmulni drm_display_helper polyval_generic usbcore cec i2c_i801 rc_core video ptp spi_intel_pci i2c_mux ghash_clmulni_intel sha512_ssse3 sha256_ssse3 sha1_ssse3 psmouse thunderbolt rtsx_pci spi_intel i2c_smbus pps_core usb_common thermal hwmon battery
[1821331.238731] wmi aesni_intel crypto_simd cryptd [last unloaded: igc]
[1821331.238737] CPU: 1 UID: 0 PID: 2457672 Comm: btrfs Tainted: G U OE 6.11.2-amd64-preempt-sysrq-20241007 #1 1a512c2db5f087f236d90ecfb30551fddcc51243
[1821331.238740] Tainted: [U]=USER, [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
[1821331.238742] Hardware name: LENOVO 20YU002JUS/20YU002JUS, BIOS N37ET49W (1.30 ) 11/15/2023
[1821331.238743] RIP: 0010:btrfs_run_delayed_refs+0x107/0x140
[1821331.238745] Code: 01 00 00 00 eb b6 e8 18 8e b7 00 31 db 89 d8 5b 5d 41 5c 41 5d 41 5e c3 cc cc cc cc 89 de 48 c7 c7 40 bb b7 86 e8 f9 0c 9f ff <0f> 0b eb d0 66 66 2e 0f 1f 84 00 00 00 00 00 66 66 2e 0f 1f 84 00
[1821331.238747] RSP: 0018:ffffaf1a6f167858 EFLAGS: 00010282
[1821331.238749] RAX: 0000000000000000 RBX: 00000000ffffff8b RCX: 0000000000000027
[1821331.238750] RDX: ffff8fcf2f2a1848 RSI: 0000000000000001 RDI: ffff8fcf2f2a1840
[1821331.238752] RBP: ffff8fb06efb8150 R08: 0000000000000000 R09: 0000000000000003
[1821331.238753] R10: ffffaf1a6f1676f8 R11: ffff8fcfaf7d5028 R12: 0000000000000000
[1821331.238754] R13: ffff8fb146731358 R14: ffff8fb146731200 R15: 0000000000000000
[1821331.238755] FS: 00007fd91883d380(0000) GS:ffff8fcf2f280000(0000) knlGS:0000000000000000
[1821331.238756] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[1821331.238757] CR2: 00007e4901001000 CR3: 00000001c3cb6004 CR4: 0000000000770ef0
[1821331.238759] PKRU: 55555554
[1821331.238760] Call Trace:
[1821331.238762] <TASK>
[1821331.238765] ? __warn+0x7c/0x140
[1821331.238769] ? btrfs_run_delayed_refs+0x107/0x140
[1821331.238771] ? report_bug+0x160/0x1c0
[1821331.238774] ? handle_bug+0x41/0x80
[1821331.238777] ? exc_invalid_op+0x15/0x100
[1821331.238780] ? asm_exc_invalid_op+0x16/0x40
[1821331.238783] ? btrfs_run_delayed_refs+0x107/0x140
[1821331.238785] ? btrfs_run_delayed_refs+0x107/0x140
[1821331.238786] btrfs_commit_transaction+0x69/0xe80
[1821331.238790] ? btrfs_update_reloc_root+0x12d/0x240
[1821331.238793] prepare_to_merge+0x4f0/0x600
[1821331.238796] relocate_block_group+0x113/0x500
[1821331.238798] btrfs_relocate_block_group+0x27a/0x440
[1821331.238800] btrfs_relocate_chunk+0x3b/0x180
[1821331.238803] btrfs_balance+0x8c1/0x1340
[1821331.238805] ? btrfs_ioctl+0x18db/0x26c0
[1821331.238811] btrfs_ioctl+0x2285/0x26c0
[1821331.238813] ? __mod_memcg_lruvec_state+0x91/0x140
[1821331.238817] ? vsnprintf+0x323/0x580
[1821331.238819] ? __slab_free+0x53/0x2c0
[1821331.238822] ? sysfs_emit+0x68/0xc0
[1821331.238826] __x64_sys_ioctl+0x90/0x100
[1821331.238830] do_syscall_64+0x69/0x140
[1821331.238832] ? __memcg_slab_free_hook+0xf3/0x140
[1821331.238835] ? __x64_sys_close+0x38/0x80
[1821331.238838] ? kmem_cache_free+0x336/0x400
[1821331.238840] ? do_syscall_64+0x75/0x140
[1821331.238842] ? ksys_read+0x63/0x100
[1821331.238845] ? __mod_memcg_lruvec_state+0x91/0x140
[1821331.238848] ? mod_objcg_state+0x19d/0x2c0
[1821331.238850] ? __memcg_slab_free_hook+0xf3/0x140
[1821331.238852] ? seq_release+0x24/0x40
[1821331.238854] ? __memcg_slab_free_hook+0xf3/0x140
[1821331.238856] ? __x64_sys_close+0x38/0x80
[1821331.238858] ? kmem_cache_free+0x336/0x400
[1821331.238860] ? clear_bhb_loop+0x45/0xc0
[1821331.238862] ? clear_bhb_loop+0x45/0xc0
[1821331.238864] ? clear_bhb_loop+0x45/0xc0
[1821331.238866] ? clear_bhb_loop+0x45/0xc0
[1821331.238867] ? clear_bhb_loop+0x45/0xc0
[1821331.238869] entry_SYSCALL_64_after_hwframe+0x76/0x7e
[1821331.238872] RIP: 0033:0x7fd9189564bb
[1821331.238874] Code: 00 48 89 44 24 18 31 c0 48 8d 44 24 60 c7 04 24 10 00 00 00 48 89 44 24 08 48 8d 44 24 20 48 89 44 24 10 b8 10 00 00 00 0f 05 <89> c2 3d 00 f0 ff ff 77 1c 48 8b 44 24 18 64 48 2b 04 25 28 00 00
[1821331.238876] RSP: 002b:00007ffd5896d7a0 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[1821331.238878] RAX: ffffffffffffffda RBX: 0000000000000003 RCX: 00007fd9189564bb
[1821331.238879] RDX: 00007ffd5896d8a8 RSI: 00000000c4009420 RDI: 0000000000000003
[1821331.238880] RBP: 0000000000000000 R08: 0000000000000073 R09: 0000000000000013
[1821331.238881] R10: 0000000000000000 R11: 0000000000000246 R12: 00007ffd5896d8a8
[1821331.238882] R13: 0000000000000000 R14: 00007ffd5896ee24 R15: 0000000000000001
[1821331.238884] </TASK>
[1821331.238885] ---[ end trace 0000000000000000 ]---
[1821331.238886] BTRFS: error (device dm-4 state A) in btrfs_run_delayed_refs:2199: errno=-117 Filesystem corrupted
--
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: BTRFS error (device dm-4): failed to run delayed ref for logical 350223581184 num_bytes 16384 type 176 action 1 ref_mod 1: -117 (kernel 6.11.2)
2025-02-06 17:07 BTRFS error (device dm-4): failed to run delayed ref for logical 350223581184 num_bytes 16384 type 176 action 1 ref_mod 1: -117 (kernel 6.11.2) Marc MERLIN
@ 2025-02-06 17:34 ` Marc MERLIN
2025-02-25 7:32 ` Marc MERLIN
2025-02-27 17:02 ` Boris Burkov
2 siblings, 0 replies; 17+ messages in thread
From: Marc MERLIN @ 2025-02-06 17:34 UTC (permalink / raw)
To: linux-btrfs
[-- Attachment #1: Type: text/plain, Size: 1211 bytes --]
On Thu, Feb 06, 2025 at 09:07:31AM -0800, Marc MERLIN wrote:
> [1821331.237716] item 0 key (350222417920 169 0) itemoff 16250 itemsize 33
> [1821331.237718] extent refs 1 gen 2907391 flags 2
> [1821331.237719] ref#0: tree block backref root 398
> (...(
> [1821331.238559] item 167 key (350225678336 169 0) itemoff 7733 itemsize 168
> [1821331.238560] extent refs 16 gen 4619087 flags 2
> [1821331.238561] ref#0: tree block backref root 398
> [1821331.238562] ref#1: shared block backref parent 737084112896
> [1821331.238563] ref#2: shared block backref parent 736609173504
> [1821331.238564] ref#3: shared block backref parent 471099588608
> [1821331.238565] ref#4: shared block backref parent 471017488384
> [1821331.238567] ref#5: shared block backref parent 470665625600
> [1821331.238568] ref#6: shared block backref parent 350806835200
> [1821331.238569] ref#7: shared block backref parent 350292066304
Sorry, I forgot to include the bzip'ed keys (big output, felt like too
much to paste, but maybe it's important)
--
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
[-- Attachment #2: keys.bz2 --]
[-- Type: application/octet-stream, Size: 6394 bytes --]
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: BTRFS error (device dm-4): failed to run delayed ref for logical 350223581184 num_bytes 16384 type 176 action 1 ref_mod 1: -117 (kernel 6.11.2)
2025-02-06 17:07 BTRFS error (device dm-4): failed to run delayed ref for logical 350223581184 num_bytes 16384 type 176 action 1 ref_mod 1: -117 (kernel 6.11.2) Marc MERLIN
2025-02-06 17:34 ` Marc MERLIN
@ 2025-02-25 7:32 ` Marc MERLIN
2025-02-25 8:07 ` Qu Wenruo
2025-02-27 17:02 ` Boris Burkov
2 siblings, 1 reply; 17+ messages in thread
From: Marc MERLIN @ 2025-02-25 7:32 UTC (permalink / raw)
To: linux-btrfs
Cc: Su Yue, linux-btrfs, Qu Wenruo, Josef Bacik, Chris Murphy,
Zygo Blaxell, Roman Mamedov, Su Yue
Got no reply, but definitely an issue, so adding a few folks to Cc
System has been working fine since then, but I have disabled nightly balance
out of fear this will happen again.
This is what I have nightly:
# I'm told that proactively rebalancing metadata may not be a good idea.
#btrfs balance start -musage=20 -v $mountpoint 2>&1 | grep -Ev "$FILTER"
# but a null rebalance should help corner cases:
btrfs balance start -musage=0 -v $mountpoint 2>&1 | grep -Ev "$FILTER"
# After metadata, let's do data:
btrfs balance start -dusage=0 -v $mountpoint 2>&1 | grep -Ev "$FILTER"
btrfs balance start -dusage=20 -v $mountpoint 2>&1 | grep -Ev "$FILTER"
Balance ended with btrfs_run_delayed_refs:2199: errno=-117 Filesystem corrupted
btrfs check says it's not
sauron:~# btrfs check /dev/mapper/pool1
Opening filesystem to check...
Checking filesystem on /dev/mapper/pool1
UUID: 4542883b-d8bc-4d7f-8a2e-944dc355dc44
[1/7] checking root items
[2/7] checking extents
[3/7] checking free space tree
[4/7] checking fs roots
[5/7] checking only csums items (without verifying data)
[6/7] checking root refs
[7/7] checking quota groups skipped (not enabled on this FS)
found 228820946944 bytes used, no error found
total csum bytes: 219270232
total tree bytes: 4539334656
total fs tree bytes: 3719593984
total extent tree bytes: 481935360
btree space waste bytes: 1075469196
file data blocks allocated: 15390875832320
referenced 290833076224
Any ideas? this obviously caused downtime, but after the btrfs check saying I'm supposedly
ok, I'm back up for now, hoping it won't happen again and hope is not a strategy :)
Is it safe to run balance again? any idea why it failed?
[1821331.015652] BTRFS info (device dm-4): balance: start -dusage=20
[1821331.015805] BTRFS info (device dm-4): relocating block group 761212698624 flags data
[1821331.090338] BTRFS info (device dm-4): found 31 extents, stage: move data extents
[1821331.237707] BTRFS info (device dm-4): leaf 471333519360 gen 4808182 total ptrs 168 free space 3533 owner 2
[1821331.237716] item 0 key (350222417920 169 0) itemoff 16250 itemsize 33
[1821331.237718] extent refs 1 gen 2907391 flags 2
[1821331.237719] ref#0: tree block backref root 398
(...(
[1821331.238559] item 167 key (350225678336 169 0) itemoff 7733 itemsize 168
[1821331.238560] extent refs 16 gen 4619087 flags 2
[1821331.238561] ref#0: tree block backref root 398
[1821331.238562] ref#1: shared block backref parent 737084112896
[1821331.238563] ref#2: shared block backref parent 736609173504
[1821331.238564] ref#3: shared block backref parent 471099588608
[1821331.238565] ref#4: shared block backref parent 471017488384
[1821331.238567] ref#5: shared block backref parent 470665625600
[1821331.238568] ref#6: shared block backref parent 350806835200
[1821331.238569] ref#7: shared block backref parent 350292066304
[1821331.238570] ref#8: shared block backref parent 349856350208
[1821331.238571] ref#9: shared block backref parent 153429573632
[1821331.238572] ref#10: shared block backref parent 153014337536
[1821331.238573] ref#11: shared block backref parent 152976048128
[1821331.238575] ref#12: shared block backref parent 152753946624
[1821331.238576] ref#13: shared block backref parent 152639225856
[1821331.238577] ref#14: shared block backref parent 50782617600
[1821331.238578] ref#15: shared block backref parent 394002432
[1821331.238579] BTRFS critical (device dm-4): adding refs to an existing tree ref, bytenr 350223581184 num_bytes 16384 root_objectid 398 slot 51
[1821331.238582] BTRFS error (device dm-4): failed to run delayed ref for logical 350223581184 num_bytes 16384 type 176 action 1 ref_mod 1: -117
[1821331.238584] ------------[ cut here ]------------
[1821331.238585] BTRFS: Transaction aborted (error -117)
[1821331.238593] WARNING: CPU: 1 PID: 2457672 at fs/btrfs/extent-tree.c:2199 btrfs_run_delayed_refs+0x107/0x140
[1821331.238599] Modules linked in: mmc_block exfat uinput rpcsec_gss_krb5 nfsv4 dns_resolver nfs netfs sg uas usb_storage nf_conntrack_netlink xfrm_user xfrm_algo xt_addrtype br_netfilter bridge stp llc xt_tcpudp xt_conntrack rfcomm snd_seq_dummy snd_hrtimer ccm overlay ipt_REJECT nf_reject_ipv4 xt_MASQUERADE xt_LOG nf_log_syslog nft_compat nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 cmac algif_hash algif_skcipher af_alg nf_tables bnep binfmt_misc uvcvideo videobuf2_vmalloc uvc videobuf2_memops btusb videobuf2_v4l2 btrtl btbcm videodev btmtk btintel videobuf2_common mc bluetooth nls_utf8 nls_cp437 vfat fat squashfs loop snd_hda_codec_hdmi iwlmvm snd_hda_codec_realtek snd_hda_codec_generic snd_soc_dmic snd_hda_scodec_component mac80211 intel_uncore_frequency snd_sof_pci_intel_tgl intel_uncore_frequency_common snd_sof_pci_intel_cnl intel_tcc_cooling snd_sof_intel_hda_generic snd_sof_intel_hda_common x86_pkg_temp_thermal snd_sof_intel_hda intel_powerclamp libarc4 snd_sof_pci snd_sof_xtensa_dsp kvm_intel
[1821331.238644] snd_soc_hdac_hda iwlwifi kvm thinkpad_acpi snd_soc_acpi_intel_match mei_hdcp mei_pxp snd_soc_acpi cfg80211 processor_thermal_device_pci_legacy nvram tpm_crb processor_thermal_device rapl platform_profile processor_thermal_wt_hint processor_thermal_rfim snd_soc_avs processor_thermal_rapl ucsi_acpi intel_rapl_common nvidiafb processor_thermal_wt_req intel_cstate think_lmi vgastate iTCO_wdt typec_ucsi snd_soc_hda_codec pcspkr processor_thermal_power_floor firmware_attributes_class wmi_bmof snd_hda_intel ee1004 iTCO_vendor_support fb_ddc typec processor_thermal_mbox mei_me roles intel_soc_dts_iosf int3403_thermal rfkill int340x_thermal_zone ac intel_pmc_core intel_vsec tpm_tis tpm_tis_core int3400_thermal pmt_telemetry acpi_pad intel_hid acpi_thermal_rel sparse_keymap pmt_class acpi_tad input_leds evdev joydev serio_raw vboxdrv(OE) soundwire_intel soundwire_cadence snd_sof_intel_hda_mlink soundwire_generic_allocation snd_sof_probes snd_sof snd_sof_utils snd_intel_dspcfg snd_intel_sdw_acpi snd_soc_skl_hda_dsp
[1821331.238682] snd_soc_intel_hda_dsp_common snd_hda_codec snd_hwdep snd_soc_hdac_hdmi snd_hda_ext_core snd_soc_core snd_compress snd_pcm_dmaengine snd_hda_core snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_midi snd_seq_midi_event snd_seq snd_timer snd_rawmidi snd_seq_device snd_ctl_led snd soundcore ac97_bus configs coretemp msr fuse efi_pstore nfsd auth_rpcgss nfs_acl lockd grace sunrpc nfnetlink ip_tables x_tables autofs4 essiv authenc dm_crypt trusted asn1_encoder tee tpm rng_core libaescfb ecdh_generic dm_mod ecc raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx sata_sil24 r8169 realtek mdio_devres libphy mii hid_generic usbhid hid i915 crct10dif_pclmul drm_buddy i2c_algo_bit rtsx_pci_sdmmc crc32_pclmul xhci_pci ttm mmc_core crc32c_intel xhci_hcd polyval_clmulni drm_display_helper polyval_generic usbcore cec i2c_i801 rc_core video ptp spi_intel_pci i2c_mux ghash_clmulni_intel sha512_ssse3 sha256_ssse3 sha1_ssse3 psmouse thunderbolt rtsx_pci spi_intel i2c_smbus pps_core usb_common thermal hwmon battery
[1821331.238731] wmi aesni_intel crypto_simd cryptd [last unloaded: igc]
[1821331.238737] CPU: 1 UID: 0 PID: 2457672 Comm: btrfs Tainted: G U OE 6.11.2-amd64-preempt-sysrq-20241007 #1 1a512c2db5f087f236d90ecfb30551fddcc51243
[1821331.238740] Tainted: [U]=USER, [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
[1821331.238742] Hardware name: LENOVO 20YU002JUS/20YU002JUS, BIOS N37ET49W (1.30 ) 11/15/2023
[1821331.238743] RIP: 0010:btrfs_run_delayed_refs+0x107/0x140
[1821331.238745] Code: 01 00 00 00 eb b6 e8 18 8e b7 00 31 db 89 d8 5b 5d 41 5c 41 5d 41 5e c3 cc cc cc cc 89 de 48 c7 c7 40 bb b7 86 e8 f9 0c 9f ff <0f> 0b eb d0 66 66 2e 0f 1f 84 00 00 00 00 00 66 66 2e 0f 1f 84 00
[1821331.238747] RSP: 0018:ffffaf1a6f167858 EFLAGS: 00010282
[1821331.238749] RAX: 0000000000000000 RBX: 00000000ffffff8b RCX: 0000000000000027
[1821331.238750] RDX: ffff8fcf2f2a1848 RSI: 0000000000000001 RDI: ffff8fcf2f2a1840
[1821331.238752] RBP: ffff8fb06efb8150 R08: 0000000000000000 R09: 0000000000000003
[1821331.238753] R10: ffffaf1a6f1676f8 R11: ffff8fcfaf7d5028 R12: 0000000000000000
[1821331.238754] R13: ffff8fb146731358 R14: ffff8fb146731200 R15: 0000000000000000
[1821331.238755] FS: 00007fd91883d380(0000) GS:ffff8fcf2f280000(0000) knlGS:0000000000000000
[1821331.238756] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[1821331.238757] CR2: 00007e4901001000 CR3: 00000001c3cb6004 CR4: 0000000000770ef0
[1821331.238759] PKRU: 55555554
[1821331.238760] Call Trace:
[1821331.238762] <TASK>
[1821331.238765] ? __warn+0x7c/0x140
[1821331.238769] ? btrfs_run_delayed_refs+0x107/0x140
[1821331.238771] ? report_bug+0x160/0x1c0
[1821331.238774] ? handle_bug+0x41/0x80
[1821331.238777] ? exc_invalid_op+0x15/0x100
[1821331.238780] ? asm_exc_invalid_op+0x16/0x40
[1821331.238783] ? btrfs_run_delayed_refs+0x107/0x140
[1821331.238785] ? btrfs_run_delayed_refs+0x107/0x140
[1821331.238786] btrfs_commit_transaction+0x69/0xe80
[1821331.238790] ? btrfs_update_reloc_root+0x12d/0x240
[1821331.238793] prepare_to_merge+0x4f0/0x600
[1821331.238796] relocate_block_group+0x113/0x500
[1821331.238798] btrfs_relocate_block_group+0x27a/0x440
[1821331.238800] btrfs_relocate_chunk+0x3b/0x180
[1821331.238803] btrfs_balance+0x8c1/0x1340
[1821331.238805] ? btrfs_ioctl+0x18db/0x26c0
[1821331.238811] btrfs_ioctl+0x2285/0x26c0
[1821331.238813] ? __mod_memcg_lruvec_state+0x91/0x140
[1821331.238817] ? vsnprintf+0x323/0x580
[1821331.238819] ? __slab_free+0x53/0x2c0
[1821331.238822] ? sysfs_emit+0x68/0xc0
[1821331.238826] __x64_sys_ioctl+0x90/0x100
[1821331.238830] do_syscall_64+0x69/0x140
[1821331.238832] ? __memcg_slab_free_hook+0xf3/0x140
[1821331.238835] ? __x64_sys_close+0x38/0x80
[1821331.238838] ? kmem_cache_free+0x336/0x400
[1821331.238840] ? do_syscall_64+0x75/0x140
[1821331.238842] ? ksys_read+0x63/0x100
[1821331.238845] ? __mod_memcg_lruvec_state+0x91/0x140
[1821331.238848] ? mod_objcg_state+0x19d/0x2c0
[1821331.238850] ? __memcg_slab_free_hook+0xf3/0x140
[1821331.238852] ? seq_release+0x24/0x40
[1821331.238854] ? __memcg_slab_free_hook+0xf3/0x140
[1821331.238856] ? __x64_sys_close+0x38/0x80
[1821331.238858] ? kmem_cache_free+0x336/0x400
[1821331.238860] ? clear_bhb_loop+0x45/0xc0
[1821331.238862] ? clear_bhb_loop+0x45/0xc0
[1821331.238864] ? clear_bhb_loop+0x45/0xc0
[1821331.238866] ? clear_bhb_loop+0x45/0xc0
[1821331.238867] ? clear_bhb_loop+0x45/0xc0
[1821331.238869] entry_SYSCALL_64_after_hwframe+0x76/0x7e
[1821331.238872] RIP: 0033:0x7fd9189564bb
[1821331.238874] Code: 00 48 89 44 24 18 31 c0 48 8d 44 24 60 c7 04 24 10 00 00 00 48 89 44 24 08 48 8d 44 24 20 48 89 44 24 10 b8 10 00 00 00 0f 05 <89> c2 3d 00 f0 ff ff 77 1c 48 8b 44 24 18 64 48 2b 04 25 28 00 00
[1821331.238876] RSP: 002b:00007ffd5896d7a0 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[1821331.238878] RAX: ffffffffffffffda RBX: 0000000000000003 RCX: 00007fd9189564bb
[1821331.238879] RDX: 00007ffd5896d8a8 RSI: 00000000c4009420 RDI: 0000000000000003
[1821331.238880] RBP: 0000000000000000 R08: 0000000000000073 R09: 0000000000000013
[1821331.238881] R10: 0000000000000000 R11: 0000000000000246 R12: 00007ffd5896d8a8
[1821331.238882] R13: 0000000000000000 R14: 00007ffd5896ee24 R15: 0000000000000001
[1821331.238884] </TASK>
[1821331.238885] ---[ end trace 0000000000000000 ]---
[1821331.238886] BTRFS: error (device dm-4 state A) in btrfs_run_delayed_refs:2199: errno=-117 Filesystem corrupted
--
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: BTRFS error (device dm-4): failed to run delayed ref for logical 350223581184 num_bytes 16384 type 176 action 1 ref_mod 1: -117 (kernel 6.11.2)
2025-02-25 7:32 ` Marc MERLIN
@ 2025-02-25 8:07 ` Qu Wenruo
2025-02-25 8:29 ` Marc MERLIN
0 siblings, 1 reply; 17+ messages in thread
From: Qu Wenruo @ 2025-02-25 8:07 UTC (permalink / raw)
To: Marc MERLIN, linux-btrfs
Cc: Su Yue, Josef Bacik, Chris Murphy, Zygo Blaxell, Roman Mamedov,
Su Yue
在 2025/2/25 18:02, Marc MERLIN 写道:
> Got no reply, but definitely an issue, so adding a few folks to Cc
> System has been working fine since then, but I have disabled nightly balance
> out of fear this will happen again.
>
> This is what I have nightly:
> # I'm told that proactively rebalancing metadata may not be a good idea.
> #btrfs balance start -musage=20 -v $mountpoint 2>&1 | grep -Ev "$FILTER"
> # but a null rebalance should help corner cases:
> btrfs balance start -musage=0 -v $mountpoint 2>&1 | grep -Ev "$FILTER"
> # After metadata, let's do data:
> btrfs balance start -dusage=0 -v $mountpoint 2>&1 | grep -Ev "$FILTER"
> btrfs balance start -dusage=20 -v $mountpoint 2>&1 | grep -Ev "$FILTER"
>
>
> Balance ended with btrfs_run_delayed_refs:2199: errno=-117 Filesystem corrupted
>
> btrfs check says it's not
> sauron:~# btrfs check /dev/mapper/pool1
> Opening filesystem to check...
> Checking filesystem on /dev/mapper/pool1
> UUID: 4542883b-d8bc-4d7f-8a2e-944dc355dc44
> [1/7] checking root items
> [2/7] checking extents
> [3/7] checking free space tree
> [4/7] checking fs roots
> [5/7] checking only csums items (without verifying data)
> [6/7] checking root refs
> [7/7] checking quota groups skipped (not enabled on this FS)
> found 228820946944 bytes used, no error found
> total csum bytes: 219270232
> total tree bytes: 4539334656
> total fs tree bytes: 3719593984
> total extent tree bytes: 481935360
> btree space waste bytes: 1075469196
> file data blocks allocated: 15390875832320
> referenced 290833076224
>
> Any ideas? this obviously caused downtime, but after the btrfs check saying I'm supposedly
> ok, I'm back up for now, hoping it won't happen again and hope is not a strategy :)
>
> Is it safe to run balance again? any idea why it failed?
>
> [1821331.015652] BTRFS info (device dm-4): balance: start -dusage=20
> [1821331.015805] BTRFS info (device dm-4): relocating block group 761212698624 flags data
> [1821331.090338] BTRFS info (device dm-4): found 31 extents, stage: move data extents
> [1821331.237707] BTRFS info (device dm-4): leaf 471333519360 gen 4808182 total ptrs 168 free space 3533 owner 2
> [1821331.237716] item 0 key (350222417920 169 0) itemoff 16250 itemsize 33
> [1821331.237718] extent refs 1 gen 2907391 flags 2
> [1821331.237719] ref#0: tree block backref root 398
> (...(
Full dmesg please.
The detailed extent tree leaf is what we really need.
Thanks,
Qu
> [1821331.238559] item 167 key (350225678336 169 0) itemoff 7733 itemsize 168
> [1821331.238560] extent refs 16 gen 4619087 flags 2
> [1821331.238561] ref#0: tree block backref root 398
> [1821331.238562] ref#1: shared block backref parent 737084112896
> [1821331.238563] ref#2: shared block backref parent 736609173504
> [1821331.238564] ref#3: shared block backref parent 471099588608
> [1821331.238565] ref#4: shared block backref parent 471017488384
> [1821331.238567] ref#5: shared block backref parent 470665625600
> [1821331.238568] ref#6: shared block backref parent 350806835200
> [1821331.238569] ref#7: shared block backref parent 350292066304
> [1821331.238570] ref#8: shared block backref parent 349856350208
> [1821331.238571] ref#9: shared block backref parent 153429573632
> [1821331.238572] ref#10: shared block backref parent 153014337536
> [1821331.238573] ref#11: shared block backref parent 152976048128
> [1821331.238575] ref#12: shared block backref parent 152753946624
> [1821331.238576] ref#13: shared block backref parent 152639225856
> [1821331.238577] ref#14: shared block backref parent 50782617600
> [1821331.238578] ref#15: shared block backref parent 394002432
> [1821331.238579] BTRFS critical (device dm-4): adding refs to an existing tree ref, bytenr 350223581184 num_bytes 16384 root_objectid 398 slot 51
> [1821331.238582] BTRFS error (device dm-4): failed to run delayed ref for logical 350223581184 num_bytes 16384 type 176 action 1 ref_mod 1: -117
> [1821331.238584] ------------[ cut here ]------------
> [1821331.238585] BTRFS: Transaction aborted (error -117)
> [1821331.238593] WARNING: CPU: 1 PID: 2457672 at fs/btrfs/extent-tree.c:2199 btrfs_run_delayed_refs+0x107/0x140
> [1821331.238599] Modules linked in: mmc_block exfat uinput rpcsec_gss_krb5 nfsv4 dns_resolver nfs netfs sg uas usb_storage nf_conntrack_netlink xfrm_user xfrm_algo xt_addrtype br_netfilter bridge stp llc xt_tcpudp xt_conntrack rfcomm snd_seq_dummy snd_hrtimer ccm overlay ipt_REJECT nf_reject_ipv4 xt_MASQUERADE xt_LOG nf_log_syslog nft_compat nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 cmac algif_hash algif_skcipher af_alg nf_tables bnep binfmt_misc uvcvideo videobuf2_vmalloc uvc videobuf2_memops btusb videobuf2_v4l2 btrtl btbcm videodev btmtk btintel videobuf2_common mc bluetooth nls_utf8 nls_cp437 vfat fat squashfs loop snd_hda_codec_hdmi iwlmvm snd_hda_codec_realtek snd_hda_codec_generic snd_soc_dmic snd_hda_scodec_component mac80211 intel_uncore_frequency snd_sof_pci_intel_tgl intel_uncore_frequency_common snd_sof_pci_intel_cnl intel_tcc_cooling snd_sof_intel_hda_generic snd_sof_intel_hda_common x86_pkg_temp_thermal snd_sof_intel_hda intel_powerclamp libarc4 snd_sof_pci snd_sof_xtensa_dsp kvm_intel
> [1821331.238644] snd_soc_hdac_hda iwlwifi kvm thinkpad_acpi snd_soc_acpi_intel_match mei_hdcp mei_pxp snd_soc_acpi cfg80211 processor_thermal_device_pci_legacy nvram tpm_crb processor_thermal_device rapl platform_profile processor_thermal_wt_hint processor_thermal_rfim snd_soc_avs processor_thermal_rapl ucsi_acpi intel_rapl_common nvidiafb processor_thermal_wt_req intel_cstate think_lmi vgastate iTCO_wdt typec_ucsi snd_soc_hda_codec pcspkr processor_thermal_power_floor firmware_attributes_class wmi_bmof snd_hda_intel ee1004 iTCO_vendor_support fb_ddc typec processor_thermal_mbox mei_me roles intel_soc_dts_iosf int3403_thermal rfkill int340x_thermal_zone ac intel_pmc_core intel_vsec tpm_tis tpm_tis_core int3400_thermal pmt_telemetry acpi_pad intel_hid acpi_thermal_rel sparse_keymap pmt_class acpi_tad input_leds evdev joydev serio_raw vboxdrv(OE) soundwire_intel soundwire_cadence snd_sof_intel_hda_mlink soundwire_generic_allocation snd_sof_probes snd_sof snd_sof_utils snd_intel_dspcfg snd_intel_sdw_acpi snd_soc_skl_hda_dsp
> [1821331.238682] snd_soc_intel_hda_dsp_common snd_hda_codec snd_hwdep snd_soc_hdac_hdmi snd_hda_ext_core snd_soc_core snd_compress snd_pcm_dmaengine snd_hda_core snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_midi snd_seq_midi_event snd_seq snd_timer snd_rawmidi snd_seq_device snd_ctl_led snd soundcore ac97_bus configs coretemp msr fuse efi_pstore nfsd auth_rpcgss nfs_acl lockd grace sunrpc nfnetlink ip_tables x_tables autofs4 essiv authenc dm_crypt trusted asn1_encoder tee tpm rng_core libaescfb ecdh_generic dm_mod ecc raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx sata_sil24 r8169 realtek mdio_devres libphy mii hid_generic usbhid hid i915 crct10dif_pclmul drm_buddy i2c_algo_bit rtsx_pci_sdmmc crc32_pclmul xhci_pci ttm mmc_core crc32c_intel xhci_hcd polyval_clmulni drm_display_helper polyval_generic usbcore cec i2c_i801 rc_core video ptp spi_intel_pci i2c_mux ghash_clmulni_intel sha512_ssse3 sha256_ssse3 sha1_ssse3 psmouse thunderbolt rtsx_pci spi_intel i2c_smbus pps_core usb_common thermal hwmon battery
> [1821331.238731] wmi aesni_intel crypto_simd cryptd [last unloaded: igc]
> [1821331.238737] CPU: 1 UID: 0 PID: 2457672 Comm: btrfs Tainted: G U OE 6.11.2-amd64-preempt-sysrq-20241007 #1 1a512c2db5f087f236d90ecfb30551fddcc51243
> [1821331.238740] Tainted: [U]=USER, [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
> [1821331.238742] Hardware name: LENOVO 20YU002JUS/20YU002JUS, BIOS N37ET49W (1.30 ) 11/15/2023
> [1821331.238743] RIP: 0010:btrfs_run_delayed_refs+0x107/0x140
> [1821331.238745] Code: 01 00 00 00 eb b6 e8 18 8e b7 00 31 db 89 d8 5b 5d 41 5c 41 5d 41 5e c3 cc cc cc cc 89 de 48 c7 c7 40 bb b7 86 e8 f9 0c 9f ff <0f> 0b eb d0 66 66 2e 0f 1f 84 00 00 00 00 00 66 66 2e 0f 1f 84 00
> [1821331.238747] RSP: 0018:ffffaf1a6f167858 EFLAGS: 00010282
> [1821331.238749] RAX: 0000000000000000 RBX: 00000000ffffff8b RCX: 0000000000000027
> [1821331.238750] RDX: ffff8fcf2f2a1848 RSI: 0000000000000001 RDI: ffff8fcf2f2a1840
> [1821331.238752] RBP: ffff8fb06efb8150 R08: 0000000000000000 R09: 0000000000000003
> [1821331.238753] R10: ffffaf1a6f1676f8 R11: ffff8fcfaf7d5028 R12: 0000000000000000
> [1821331.238754] R13: ffff8fb146731358 R14: ffff8fb146731200 R15: 0000000000000000
> [1821331.238755] FS: 00007fd91883d380(0000) GS:ffff8fcf2f280000(0000) knlGS:0000000000000000
> [1821331.238756] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [1821331.238757] CR2: 00007e4901001000 CR3: 00000001c3cb6004 CR4: 0000000000770ef0
> [1821331.238759] PKRU: 55555554
> [1821331.238760] Call Trace:
> [1821331.238762] <TASK>
> [1821331.238765] ? __warn+0x7c/0x140
> [1821331.238769] ? btrfs_run_delayed_refs+0x107/0x140
> [1821331.238771] ? report_bug+0x160/0x1c0
> [1821331.238774] ? handle_bug+0x41/0x80
> [1821331.238777] ? exc_invalid_op+0x15/0x100
> [1821331.238780] ? asm_exc_invalid_op+0x16/0x40
> [1821331.238783] ? btrfs_run_delayed_refs+0x107/0x140
> [1821331.238785] ? btrfs_run_delayed_refs+0x107/0x140
> [1821331.238786] btrfs_commit_transaction+0x69/0xe80
> [1821331.238790] ? btrfs_update_reloc_root+0x12d/0x240
> [1821331.238793] prepare_to_merge+0x4f0/0x600
> [1821331.238796] relocate_block_group+0x113/0x500
> [1821331.238798] btrfs_relocate_block_group+0x27a/0x440
> [1821331.238800] btrfs_relocate_chunk+0x3b/0x180
> [1821331.238803] btrfs_balance+0x8c1/0x1340
> [1821331.238805] ? btrfs_ioctl+0x18db/0x26c0
> [1821331.238811] btrfs_ioctl+0x2285/0x26c0
> [1821331.238813] ? __mod_memcg_lruvec_state+0x91/0x140
> [1821331.238817] ? vsnprintf+0x323/0x580
> [1821331.238819] ? __slab_free+0x53/0x2c0
> [1821331.238822] ? sysfs_emit+0x68/0xc0
> [1821331.238826] __x64_sys_ioctl+0x90/0x100
> [1821331.238830] do_syscall_64+0x69/0x140
> [1821331.238832] ? __memcg_slab_free_hook+0xf3/0x140
> [1821331.238835] ? __x64_sys_close+0x38/0x80
> [1821331.238838] ? kmem_cache_free+0x336/0x400
> [1821331.238840] ? do_syscall_64+0x75/0x140
> [1821331.238842] ? ksys_read+0x63/0x100
> [1821331.238845] ? __mod_memcg_lruvec_state+0x91/0x140
> [1821331.238848] ? mod_objcg_state+0x19d/0x2c0
> [1821331.238850] ? __memcg_slab_free_hook+0xf3/0x140
> [1821331.238852] ? seq_release+0x24/0x40
> [1821331.238854] ? __memcg_slab_free_hook+0xf3/0x140
> [1821331.238856] ? __x64_sys_close+0x38/0x80
> [1821331.238858] ? kmem_cache_free+0x336/0x400
> [1821331.238860] ? clear_bhb_loop+0x45/0xc0
> [1821331.238862] ? clear_bhb_loop+0x45/0xc0
> [1821331.238864] ? clear_bhb_loop+0x45/0xc0
> [1821331.238866] ? clear_bhb_loop+0x45/0xc0
> [1821331.238867] ? clear_bhb_loop+0x45/0xc0
> [1821331.238869] entry_SYSCALL_64_after_hwframe+0x76/0x7e
> [1821331.238872] RIP: 0033:0x7fd9189564bb
> [1821331.238874] Code: 00 48 89 44 24 18 31 c0 48 8d 44 24 60 c7 04 24 10 00 00 00 48 89 44 24 08 48 8d 44 24 20 48 89 44 24 10 b8 10 00 00 00 0f 05 <89> c2 3d 00 f0 ff ff 77 1c 48 8b 44 24 18 64 48 2b 04 25 28 00 00
> [1821331.238876] RSP: 002b:00007ffd5896d7a0 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
> [1821331.238878] RAX: ffffffffffffffda RBX: 0000000000000003 RCX: 00007fd9189564bb
> [1821331.238879] RDX: 00007ffd5896d8a8 RSI: 00000000c4009420 RDI: 0000000000000003
> [1821331.238880] RBP: 0000000000000000 R08: 0000000000000073 R09: 0000000000000013
> [1821331.238881] R10: 0000000000000000 R11: 0000000000000246 R12: 00007ffd5896d8a8
> [1821331.238882] R13: 0000000000000000 R14: 00007ffd5896ee24 R15: 0000000000000001
> [1821331.238884] </TASK>
> [1821331.238885] ---[ end trace 0000000000000000 ]---
> [1821331.238886] BTRFS: error (device dm-4 state A) in btrfs_run_delayed_refs:2199: errno=-117 Filesystem corrupted
>
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: BTRFS error (device dm-4): failed to run delayed ref for logical 350223581184 num_bytes 16384 type 176 action 1 ref_mod 1: -117 (kernel 6.11.2)
2025-02-25 8:07 ` Qu Wenruo
@ 2025-02-25 8:29 ` Marc MERLIN
2025-02-25 8:46 ` Qu Wenruo
0 siblings, 1 reply; 17+ messages in thread
From: Marc MERLIN @ 2025-02-25 8:29 UTC (permalink / raw)
To: Qu Wenruo
Cc: linux-btrfs, Su Yue, Josef Bacik, Chris Murphy, Zygo Blaxell,
Roman Mamedov, Su Yue
On Tue, Feb 25, 2025 at 06:37:40PM +1030, Qu Wenruo wrote:
> Full dmesg please.
>
> The detailed extent tree leaf is what we really need.
Sure thing, it was posted in the first message but too big to put inside the message body without bzip2
https://www.spinics.net/lists/linux-btrfs/msg152512.html
https://www.spinics.net/lists/linux-btrfs/attachments/biniWUoCw198B.bin (keys.bz2)
Marc
--
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: BTRFS error (device dm-4): failed to run delayed ref for logical 350223581184 num_bytes 16384 type 176 action 1 ref_mod 1: -117 (kernel 6.11.2)
2025-02-25 8:29 ` Marc MERLIN
@ 2025-02-25 8:46 ` Qu Wenruo
2025-02-25 9:18 ` Marc MERLIN
0 siblings, 1 reply; 17+ messages in thread
From: Qu Wenruo @ 2025-02-25 8:46 UTC (permalink / raw)
To: Marc MERLIN, Qu Wenruo
Cc: linux-btrfs, Su Yue, Josef Bacik, Chris Murphy, Zygo Blaxell,
Roman Mamedov, Su Yue
在 2025/2/25 18:59, Marc MERLIN 写道:
> On Tue, Feb 25, 2025 at 06:37:40PM +1030, Qu Wenruo wrote:
>> Full dmesg please.
>>
>> The detailed extent tree leaf is what we really need.
>
> Sure thing, it was posted in the first message but too big to put inside the message body without bzip2
> https://www.spinics.net/lists/linux-btrfs/msg152512.html
>
> https://www.spinics.net/lists/linux-btrfs/attachments/biniWUoCw198B.bin (keys.bz2)
>
> Marc
This really looks like something went wrong for this slot:
item 51 key (350223581184 169 1) itemoff 13397 itemsize 393
extent refs 41 gen 4807808 flags 2
ref#0: tree block backref root 479807
ref#1: tree block backref root 479806
ref#2: tree block backref root 479805
ref#3: tree block backref root 479804
ref#4: tree block backref root 479803
ref#5: tree block backref root 479802
ref#6: tree block backref root 479801
ref#7: tree block backref root 479800
ref#8: tree block backref root 479799
ref#9: tree block backref root 479798
ref#10: tree block backref root 479795
ref#11: tree block backref root 479786
ref#12: tree block backref root 479773
ref#13: tree block backref root 398
ref#14: shared block backref parent 737072365568
ref#15: shared block backref parent 736782532608
ref#16: shared block backref parent 471272603648
ref#17: shared block backref parent 471272472576
ref#18: shared block backref parent 471272341504
ref#19: shared block backref parent 471272194048
ref#20: shared block backref parent 471272062976
ref#21: shared block backref parent 471271964672
ref#22: shared block backref parent 471271735296
ref#23: shared block backref parent 471271587840
ref#24: shared block backref parent 471271129088
ref#25: shared block backref parent 471270735872
ref#26: shared block backref parent 471270522880
ref#27: shared block backref parent 471270391808
ref#28: shared block backref parent 471270293504
ref#29: shared block backref parent 471270096896
ref#30: shared block backref parent 471013490688
ref#31: shared block backref parent 470792732672
ref#32: shared block backref parent 350684020736
ref#33: shared block backref parent 350589992960
ref#34: shared block backref parent 350453350400
ref#35: shared block backref parent 350407852032
ref#36: shared block backref parent 349158178816
ref#37: shared block backref parent 349102440448
ref#38: shared block backref parent 153059819520
ref#39: shared block backref parent 50772246528
ref#40: shared block backref parent 50765955072
Ref#13 is the root that mentioned has the existing one.
I have no idea why it shows up like this.
But since the fs passes btrfs check, mind to dump the following tree block?
# btrfs ins dump-tree -t extent <device> | grep "(350223581184 " -A 50
I want to make sure if the ref 398 exists on disk, or it's generated at
runtime.
And as usual, memtest.
I believe your machine is a ThinkPad P17 gen2 with a mobile Xeon, with
DDR4 ECC memory support, but I'm not sure if your memory sticks have ECC.
Just as a precaution, please run a memtest (I know it will be painfully
slow, so please only run it after the above dump is taken).
Thanks,
Qu
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: BTRFS error (device dm-4): failed to run delayed ref for logical 350223581184 num_bytes 16384 type 176 action 1 ref_mod 1: -117 (kernel 6.11.2)
2025-02-25 8:46 ` Qu Wenruo
@ 2025-02-25 9:18 ` Marc MERLIN
2025-02-25 9:22 ` Qu Wenruo
2025-02-25 9:25 ` Roman Mamedov
0 siblings, 2 replies; 17+ messages in thread
From: Marc MERLIN @ 2025-02-25 9:18 UTC (permalink / raw)
To: Qu Wenruo
Cc: Qu Wenruo, linux-btrfs, Josef Bacik, Chris Murphy, Zygo Blaxell,
Roman Mamedov, Su Yue
On Tue, Feb 25, 2025 at 07:16:31PM +1030, Qu Wenruo wrote:
> Ref#13 is the root that mentioned has the existing one.
>
> I have no idea why it shows up like this.
>
> But since the fs passes btrfs check, mind to dump the following tree block?
>
> # btrfs ins dump-tree -t extent <device> | grep "(350223581184 " -A 50
>
> I want to make sure if the ref 398 exists on disk, or it's generated at
> runtime.
sauron:~# btrfs ins dump-tree -t extent /dev/mapper/pool1 | grep "350223581184 " -A 50
sauron:~#
Mmmmh, so 350223581184 is gone now since it's been 20 days already since
my original post.
I can try to re-enable balance and see if things crash or not.
> I believe your machine is a ThinkPad P17 gen2 with a mobile Xeon, with DDR4
> ECC memory support, but I'm not sure if your memory sticks have ECC.
model name : Intel(R) Xeon(R) W-11855M CPU @ 3.20GHz
sauron:~# dmesg | grep -i EDAC
[ 1.293722] EDAC MC: Ver: 3.0.0
sauron:~# dmidecode -t memory | grep Width
Total Width: 64 bits
Data Width: 64 bits
Total Width: 64 bits
Data Width: 64 bits
Total Width: 64 bits
Data Width: 64 bits
Total Width: 64 bits
Data Width: 64 bits
That seems do say I do not have ECC or width would be 72 from what I
read :-/
> Just as a precaution, please run a memtest (I know it will be painfully
> slow, so please only run it after the above dump is taken).
I do not have the system with me right now and I think memtest cannot be
run from inside linux, correct?
If so, I can do this when I get home and can reboot the system.
Or is there a version I can run from inside linux?
Marc
--
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: BTRFS error (device dm-4): failed to run delayed ref for logical 350223581184 num_bytes 16384 type 176 action 1 ref_mod 1: -117 (kernel 6.11.2)
2025-02-25 9:18 ` Marc MERLIN
@ 2025-02-25 9:22 ` Qu Wenruo
2025-02-27 14:11 ` Marc MERLIN
2025-02-25 9:25 ` Roman Mamedov
1 sibling, 1 reply; 17+ messages in thread
From: Qu Wenruo @ 2025-02-25 9:22 UTC (permalink / raw)
To: Marc MERLIN, Qu Wenruo
Cc: linux-btrfs, Josef Bacik, Chris Murphy, Zygo Blaxell,
Roman Mamedov, Su Yue
在 2025/2/25 19:48, Marc MERLIN 写道:
> On Tue, Feb 25, 2025 at 07:16:31PM +1030, Qu Wenruo wrote:
>> Ref#13 is the root that mentioned has the existing one.
>>
>> I have no idea why it shows up like this.
>>
>> But since the fs passes btrfs check, mind to dump the following tree block?
>>
>> # btrfs ins dump-tree -t extent <device> | grep "(350223581184 " -A 50
>>
>> I want to make sure if the ref 398 exists on disk, or it's generated at
>> runtime.
>
> sauron:~# btrfs ins dump-tree -t extent /dev/mapper/pool1 | grep "350223581184 " -A 50
> sauron:~#
>
> Mmmmh, so 350223581184 is gone now since it's been 20 days already since
> my original post.
> I can try to re-enable balance and see if things crash or not.
>
>> I believe your machine is a ThinkPad P17 gen2 with a mobile Xeon, with DDR4
>> ECC memory support, but I'm not sure if your memory sticks have ECC.
>
> model name : Intel(R) Xeon(R) W-11855M CPU @ 3.20GHz
> sauron:~# dmesg | grep -i EDAC
> [ 1.293722] EDAC MC: Ver: 3.0.0
> sauron:~# dmidecode -t memory | grep Width
> Total Width: 64 bits
> Data Width: 64 bits
> Total Width: 64 bits
> Data Width: 64 bits
> Total Width: 64 bits
> Data Width: 64 bits
> Total Width: 64 bits
> Data Width: 64 bits
>
> That seems do say I do not have ECC or width would be 72 from what I
> read :-/
>
>> Just as a precaution, please run a memtest (I know it will be painfully
>> slow, so please only run it after the above dump is taken).
>
> I do not have the system with me right now and I think memtest cannot be
> run from inside linux, correct?
> If so, I can do this when I get home and can reboot the system.
>
> Or is there a version I can run from inside linux?
Memtester can do that, but it always leaves some small memory left untested.
But that should still be better than nothing.
Thanks,
Qu
>
> Marc
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: BTRFS error (device dm-4): failed to run delayed ref for logical 350223581184 num_bytes 16384 type 176 action 1 ref_mod 1: -117 (kernel 6.11.2)
2025-02-25 9:18 ` Marc MERLIN
2025-02-25 9:22 ` Qu Wenruo
@ 2025-02-25 9:25 ` Roman Mamedov
1 sibling, 0 replies; 17+ messages in thread
From: Roman Mamedov @ 2025-02-25 9:25 UTC (permalink / raw)
To: Marc MERLIN
Cc: Qu Wenruo, Qu Wenruo, linux-btrfs, Josef Bacik, Chris Murphy,
Zygo Blaxell, Su Yue
On Tue, 25 Feb 2025 01:18:57 -0800
Marc MERLIN <marc@merlins.org> wrote:
> I do not have the system with me right now and I think memtest cannot be
> run from inside linux, correct?
> If so, I can do this when I get home and can reboot the system.
>
> Or is there a version I can run from inside linux?
There is memtester: https://pyropus.ca./software/memtester/
But it can't test 100% of the RAM, only the free userspace area that it can
allocate. You should close down most apps and then give it (as a parameter) as
high amount as possible, without it getting killed by OOM. It could catch very
obvious faults, and it is a good "something to do" for now :) but for the
complete rigorous test you'll need the baremetal memtest86+.
--
With respect,
Roman
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: BTRFS error (device dm-4): failed to run delayed ref for logical 350223581184 num_bytes 16384 type 176 action 1 ref_mod 1: -117 (kernel 6.11.2)
2025-02-25 9:22 ` Qu Wenruo
@ 2025-02-27 14:11 ` Marc MERLIN
2025-02-27 15:05 ` Marc MERLIN
2025-02-28 13:10 ` Marc MERLIN
0 siblings, 2 replies; 17+ messages in thread
From: Marc MERLIN @ 2025-02-27 14:11 UTC (permalink / raw)
To: Qu Wenruo
Cc: Qu Wenruo, linux-btrfs, Josef Bacik, Chris Murphy, Zygo Blaxell,
Roman Mamedov, Su Yue
On Tue, Feb 25, 2025 at 07:52:11PM +1030, Qu Wenruo wrote:
>
>
> 在 2025/2/25 19:48, Marc MERLIN 写道:
> > On Tue, Feb 25, 2025 at 07:16:31PM +1030, Qu Wenruo wrote:
> > > Ref#13 is the root that mentioned has the existing one.
> > >
> > > I have no idea why it shows up like this.
> > >
> > > But since the fs passes btrfs check, mind to dump the following tree block?
> > >
> > > # btrfs ins dump-tree -t extent <device> | grep "(350223581184 " -A 50
> > >
> > > I want to make sure if the ref 398 exists on disk, or it's generated at
> > > runtime.
> >
> > sauron:~# btrfs ins dump-tree -t extent /dev/mapper/pool1 | grep "350223581184 " -A 50
> > sauron:~#
> >
> > Mmmmh, so 350223581184 is gone now since it's been 20 days already since
> > my original post.
> > I can try to re-enable balance and see if things crash or not.
Ok, I didn't run memtest last night, but ran a new balance to get new
offsets so can you can tell me which dump tree you'd like so you
hopefully get a non 0 result this time
I'll reboot the system after that, btrfs check the FS, wait for your dumptree
command, nad will run memtest overnight
Note that the source code seems to have an extra newline after itemsize
that could be removed?
BTRFS info (device dm-0): balance: ended with status: 0
BTRFS info (device dm-0): balance: start -dusage=0
BTRFS info (device dm-0): balance: ended with status: 0
BTRFS info (device dm-0): balance: start -dusage=20
BTRFS info (device dm-0): balance: ended with status: 0
BTRFS info (device dm-0): scrub: started on devid 1
BTRFS info (device dm-0): scrub: finished on devid 1
[9with status: 0
BTRFS info (device dm-4): balance: resume -dusage=20
BTRFS info (device dm-4): relocating block group
[9722893537280 flags data
BTRFS info (device dm-4): found 2 extents, stage: move
[9data extents
BTRFS info (device dm-4): leaf 780107776 gen 4865726
[9total ptrs 169 free space 2341 owner 2
item 0 key (470809247744 169 0) itemoff 16250 itemsize
[933
extent refs 1 gen 4541732 flags 2
ref#0: tree block backref root 398
item 1 key (470809264128 169 1) itemoff 15551 itemsize
[9699
extent refs 75 gen 4541732 flags 2
ref#0: tree block backref root 486052
ref#1: tree block backref root 486051
ref#2: tree block backref root 486050
ref#3: tree block backref root 486049
ref#4: tree block backref root 486048
ref#5: tree block backref root 486047
ref#6: tree block backref root 486046
ref#7: tree block backref root 486045
ref#8: tree block backref root 486044
ref#9: tree block backref root 486043
ref#10: tree block backref root 486035
ref#11: tree block backref root 486031
ref#12: tree block backref root 486018
ref#13: tree block backref root 486005
ref#14: tree block backref root 485992
ref#15: tree block backref root 485979
ref#16: tree block backref root 485966
ref#17: tree block backref root 485953
ref#18: tree block backref root 485721
ref#19: tree block backref root 485408
ref#20: tree block backref root 484779
ref#21: tree block backref root 482587
ref#22: tree block backref root 480469
ref#23: tree block backref root 478601
ref#24: tree block backref root 398
ref#25: shared block backref parent 737472331776
ref#26: shared block backref parent 737441284096
ref#27: shared block backref parent 737440071680
ref#28: shared block backref parent 737100906496
ref#29: shared block backref parent 737056882688
ref#30: shared block backref parent 736788856832
ref#31: shared block backref parent 736692797440
ref#32: shared block backref parent 736613449728
ref#33: shared block backref parent 471171776512
ref#34: shared block backref parent 470901473280
ref#35: shared block backref parent 470809182208
ref#36: shared block backref parent 470492151808
ref#37: shared block backref parent 350754684928
ref#38: shared block backref parent 350409670656
ref#39: shared block backref parent 350370103296
ref#40: shared block backref parent 349999218688
ref#41: shared block backref parent 349334798336
ref#42: shared block backref parent 349189128192
ref#43: shared block backref parent 153070862336
ref#44: shared block backref parent 153039044608
ref#45: shared block backref parent 152908038144
ref#46: shared block backref parent 152557928448
ref#47: shared block backref parent 51518570496
ref#48: shared block backref parent 51338706944
ref#49: shared block backref parent 509575168
ref#50: shared block backref parent 509460480
ref#51: shared block backref parent 509280256
ref#52: shared block backref parent 509198336
ref#53: shared block backref parent 509149184
ref#54: shared block backref parent 509034496
ref#55: shared block backref parent 508805120
ref#56: shared block backref parent 508575744
ref#57: shared block backref parent 508411904
ref#58: shared block backref parent 508297216
ref#59: shared block backref parent 507887616
ref#60: shared block backref parent 507772928
ref#61: shared block backref parent 505888768
ref#62: shared block backref parent 505446400
ref#63: shared block backref parent 505184256
ref#64: shared block backref parent 504774656
ref#65: shared block backref parent 504430592
ref#66: shared block backref parent 503742464
ref#67: shared block backref parent 502923264
ref#68: shared block backref parent 502415360
ref#69: shared block backref parent 496648192
ref#70: shared block backref parent 496418816
ref#71: shared block backref parent 495665152
ref#72: shared block backref parent 495190016
ref#73: shared block backref parent 494993408
ref#74: shared block backref parent 291684352
item 2 key (470809280512 169 0) itemoff 15500 itemsize
[951
extent refs 3 gen 3261729 flags 2
ref#0: tree block backref root 398
ref#1: shared block backref parent 737199292416
ref#2: shared block backref parent 349886251008
item 3 key (470809296896 169 0) itemoff 15467 itemsize
[933
extent refs 1 gen 4541732 flags 2
ref#0: tree block backref root 398
item 4 key (470809313280 169 0) itemoff 15434 itemsize
[933
extent refs 1 gen 3786399 flags 2
ref#0: tree block backref root 372498
item 5 key (470809329664 169 0) itemoff 15401 itemsize
[933
extent refs 1 gen 4865700 flags 2
ref#0: tree block backref root 486050
item 6 key (470809346048 169 0) itemoff 15161 itemsize
[9240
extent refs 24 gen 4813689 flags 2
ref#0: tree block backref root 398
ref#1: shared block backref parent 737488977920
ref#2: shared block backref parent 737398013952
ref#3: shared block backref parent 737220558848
ref#4: shared block backref parent 736785645568
ref#5: shared block backref parent 736770834432
ref#6: shared block backref parent 736759595008
ref#7: shared block backref parent 736698220544
ref#8: shared block backref parent 471391174656
ref#9: shared block backref parent 471054860288
ref#10: shared block backref parent 470850879488
ref#11: shared block backref parent 470765010944
ref#12: shared block backref parent 470477684736
ref#13: shared block backref parent 350667014144
ref#14: shared block backref parent 350431625216
ref#15: shared block backref parent 350351753216
ref#16: shared block backref parent 349566025728
ref#17: shared block backref parent 349358342144
ref#18: shared block backref parent 153255591936
ref#19: shared block backref parent 153023758336
ref#20: shared block backref parent 153003704320
ref#21: shared block backref parent 152648957952
ref#22: shared block backref parent 51431325696
ref#23: shared block backref parent 51240222720
item 7 key (470809362432 169 1) itemoff 15128 itemsize
[933
extent refs 1 gen 4865700 flags 2
ref#0: tree block backref root 486050
item 8 key (470809378816 169 0) itemoff 15095 itemsize
[933
extent refs 1 gen 4865700 flags 2
ref#0: tree block backref root 486050
item 9 key (470809395200 169 1) itemoff 15062 itemsize
[933
extent refs 1 gen 4865700 flags 2
ref#0: tree block backref root 486050
item 10 key (470809411584 169 0) itemoff 15029 itemsize
[933
extent refs 1 gen 4865700 flags 2
ref#0: tree block backref root 486050
item 11 key (470809427968 169 1) itemoff 14996 itemsize
33
extent refs 1 gen 4865700 flags 2
ref#0: tree block backref root 486050
item 12 key (470809444352 169 0) itemoff 14963 itemsize
33
extent refs 1 gen 4865700 flags 2
ref#0: tree block backref root 486050
item 13 key (470809460736 169 0) itemoff 14930 itemsize
33
extent refs 1 gen 4541732 flags 2
ref#0: tree block backref root 398
item 14 key (470809477120 169 1) itemoff 14897 itemsize
33
extent refs 1 gen 4865700 flags 2
ref#0: tree block backref root 486050
item 15 key (470809493504 169 0) itemoff 14864 itemsize
33
extent refs 1 gen 4865700 flags 2
ref#0: tree block backref root 486050
item 16 key (470809509888 169 0) itemoff 14831 itemsize
33
extent refs 1 gen 4541732 flags 2
ref#0: tree block backref root 398
item 17 key (470809526272 169 0) itemoff 14798 itemsize
33
extent refs 1 gen 4865700 flags 2
ref#0: tree block backref root 486050
item 18 key (470809542656 169 1) itemoff 14765 itemsize
33
extent refs 1 gen 4865700 flags 2
ref#0: tree block backref root 486050
item 19 key (470809559040 169 0) itemoff 14732 itemsize
33
extent refs 1 gen 4865700 flags 2
ref#0: tree block backref root 486050
item 20 key (470809575424 169 0) itemoff 14699 itemsize
33
extent refs 1 gen 4541732 flags 2
ref#0: tree block backref root 398
item 21 key (470809591808 169 0) itemoff 14666 itemsize
33
extent refs 1 gen 3500792 flags 2
ref#0: tree block backref root 398
item 22 key (470809608192 169 0) itemoff 14633 itemsize
33
extent refs 1 gen 4865700 flags 2
ref#0: tree block backref root 486050
item 23 key (470809624576 169 1) itemoff 14600 itemsize
33
extent refs 1 gen 4865700 flags 2
ref#0: tree block backref root 486050
item 24 key (470809640960 169 0) itemoff 14567 itemsize
33
extent refs 1 gen 4814306 flags 258
ref#0: shared block backref parent 152614600704
item 25 key (470809657344 169 0) itemoff 14534 itemsize
33
extent refs 1 gen 2940687 flags 2
ref#0: tree block backref root 398
item 26 key (470809673728 169 0) itemoff 14501 itemsize
33
extent refs 1 gen 4865700 flags 2
ref#0: tree block backref root 486050
item 27 key (470809690112 169 0) itemoff 14468 itemsize
33
extent refs 1 gen 4541732 flags 2
ref#0: tree block backref root 398
item 28 key (470809706496 169 0) itemoff 14435 itemsize
33
extent refs 1 gen 4865700 flags 2
ref#0: tree block backref root 486050
item 29 key (470809722880 169 1) itemoff 14402 itemsize
33
extent refs 1 gen 4865700 flags 2
ref#0: tree block backref root 486050
item 30 key (470809739264 169 0) itemoff 14369 itemsize
33
extent refs 1 gen 4796053 flags 258
ref#0: shared block backref parent 152641617920
item 31 key (470809755648 169 0) itemoff 14336 itemsize
33
extent refs 1 gen 4865700 flags 2
ref#0: tree block backref root 486050
item 32 key (470809772032 169 0) itemoff 14276 itemsize
60
extent refs 4 gen 4471163 flags 258
ref#0: shared block backref parent 350507761664
ref#1: shared block backref parent 350441930752
ref#2: shared block backref parent 153473925120
ref#3: shared block backref parent 152892456960
item 33 key (470809788416 169 0) itemoff 14243 itemsize
33
extent refs 1 gen 3447280 flags 2
ref#0: tree block backref root 398
item 34 key (470809804800 169 0) itemoff 14210 itemsize
33
extent refs 1 gen 4865700 flags 2
ref#0: tree block backref root 486050
item 35 key (470809821184 169 1) itemoff 14177 itemsize
33
extent refs 1 gen 4865700 flags 2
ref#0: tree block backref root 486050
item 36 key (470809837568 169 0) itemoff 14144 itemsize
33
extent refs 1 gen 4865700 flags 2
ref#0: tree block backref root 486050
item 37 key (470809853952 169 1) itemoff 13895 itemsize
249
extent refs 25 gen 4541732 flags 2
ref#0: tree block backref root 486052
ref#1: tree block backref root 486051
ref#2: tree block backref root 486050
ref#3: tree block backref root 486049
ref#4: tree block backref root 486048
ref#5: tree block backref root 486047
ref#6: tree block backref root 486046
ref#7: tree block backref root 486045
ref#8: tree block backref root 486044
ref#9: tree block backref root 486043
ref#10: tree block backref root 486035
ref#11: tree block backref root 486031
ref#12: tree block backref root 486018
ref#13: tree block backref root 486005
ref#14: tree block backref root 485992
ref#15: tree block backref root 485979
ref#16: tree block backref root 485966
ref#17: tree block backref root 485953
ref#18: tree block backref root 485721
ref#19: tree block backref root 485408
ref#20: tree block backref root 484779
ref#21: tree block backref root 482587
ref#22: tree block backref root 480469
ref#23: tree block backref root 478601
ref#24: tree block backref root 398
item 38 key (470809870336 169 0) itemoff 13862 itemsize
33
extent refs 1 gen 4865000 flags 258
ref#0: tree block backref root 485979
item 39 key (470809903104 169 0) itemoff 13829 itemsize
33
extent refs 1 gen 4865000 flags 258
ref#0: shared block backref parent 153004752896
item 40 key (470809985024 169 0) itemoff 13796 itemsize
33
extent refs 1 gen 3786399 flags 2
ref#0: tree block backref root 372498
item 41 key (470810001408 169 0) itemoff 13745 itemsize
51
extent refs 3 gen 3261729 flags 2
ref#0: tree block backref root 398
ref#1: shared block backref parent 737199292416
ref#2: shared block backref parent 349886251008
item 42 key (470810034176 169 0) itemoff 13712 itemsize
33
extent refs 1 gen 3529096 flags 2
ref#0: tree block backref root 398
item 43 key (470810116096 169 0) itemoff 13679 itemsize
33
extent refs 1 gen 3529096 flags 2
ref#0: tree block backref root 398
item 44 key (470810165248 169 0) itemoff 13601 itemsize
78
extent refs 6 gen 4838387 flags 2
ref#0: tree block backref root 398
ref#1: shared block backref parent 470774349824
ref#2: shared block backref parent 350775836672
ref#3: shared block backref parent 51398656000
ref#4: shared block backref parent 51369590784
ref#5: shared block backref parent 759103488
item 45 key (470810230784 169 0) itemoff 13550 itemsize
51
extent refs 3 gen 3261729 flags 2
ref#0: tree block backref root 398
ref#1: shared block backref parent 737199292416
ref#2: shared block backref parent 349886251008
item 46 key (470810247168 169 0) itemoff 13427 itemsize
123
extent refs 11 gen 4378843 flags 2
ref#0: tree block backref root 398
ref#1: shared block backref parent 471317004288
ref#2: shared block backref parent 471307550720
ref#3: shared block backref parent 471070687232
ref#4: shared block backref parent 351045566464
ref#5: shared block backref parent 153558728704
ref#6: shared block backref parent 152896454656
ref#7: shared block backref parent 152873271296
ref#8: shared block backref parent 152580521984
ref#9: shared block backref parent 50820874240
ref#10: shared block backref parent 1095565312
item 47 key (470810263552 169 0) itemoff 13358 itemsize
69
extent refs 5 gen 4838387 flags 2
ref#0: tree block backref root 398
ref#1: shared block backref parent 153317752832
ref#2: shared block backref parent 51398606848
ref#3: shared block backref parent 51369246720
ref#4: shared block backref parent 759021568
item 48 key (470810279936 169 0) itemoff 13325 itemsize
33
extent refs 1 gen 3790158 flags 2
ref#0: tree block backref root 398
item 49 key (470810296320 169 0) itemoff 13292 itemsize
33
extent refs 1 gen 4814184 flags 258
ref#0: shared block backref parent 737356562432
item 50 key (470810361856 169 0) itemoff 13259 itemsize
33
extent refs 1 gen 4864449 flags 2
ref#0: tree block backref root 2
item 51 key (470810394624 169 0) itemoff 13226 itemsize
33
extent refs 1 gen 4864449 flags 2
ref#0: tree block backref root 2
item 52 key (470810411008 169 0) itemoff 13193 itemsize
33
extent refs 1 gen 4864449 flags 2
ref#0: tree block backref root 2
item 53 key (470810427392 169 0) itemoff 13160 itemsize
33
extent refs 1 gen 4837772 flags 258
ref#0: shared block backref parent 737315373056
item 54 key (470810443776 169 0) itemoff 13127 itemsize
33
extent refs 1 gen 4837772 flags 258
ref#0: shared block backref parent 737315373056
item 55 key (470810509312 169 0) itemoff 12932 itemsize
195
extent refs 19 gen 3934086 flags 2
ref#0: tree block backref root 398
ref#1: shared block backref parent 737641054208
ref#2: shared block backref parent 737489911808
ref#3: shared block backref parent 737216643072
ref#4: shared block backref parent 737045807104
ref#5: shared block backref parent 736820330496
ref#6: shared block backref parent 736769032192
ref#7: shared block backref parent 471276716032
ref#8: shared block backref parent 351090245632
ref#9: shared block backref parent 350562500608
ref#10: shared block backref parent 350442569728
ref#11: shared block backref parent 350177230848
ref#12: shared block backref parent 349864771584
ref#13: shared block backref parent 349796909056
ref#14: shared block backref parent 349431185408
ref#15: shared block backref parent 153273909248
ref#16: shared block backref parent 152756305920
ref#17: shared block backref parent 51273728000
ref#18: shared block backref parent 50574934016
item 56 key (470810542080 169 0) itemoff 12872 itemsize
60
extent refs 4 gen 3723805 flags 2
ref#0: tree block backref root 398
ref#1: shared block backref parent 737008451584
ref#2: shared block backref parent 736887128064
ref#3: shared block backref parent 50967035904
item 57 key (470810574848 169 0) itemoff 12839 itemsize
33
extent refs 1 gen 4814304 flags 258
ref#0: shared block backref parent 152773263360
item 58 key (470810624000 169 0) itemoff 12806 itemsize
33
extent refs 1 gen 3786399 flags 2
ref#0: tree block backref root 372498
item 59 key (470810640384 169 0) itemoff 12773 itemsize
33
extent refs 1 gen 3786399 flags 2
ref#0: tree block backref root 372498
item 60 key (470810722304 169 0) itemoff 12740 itemsize
33
extent refs 1 gen 4810538 flags 258
ref#0: shared block backref parent 470348480512
item 61 key (470810771456 169 0) itemoff 12707 itemsize
33
extent refs 1 gen 4833910 flags 258
ref#0: shared block backref parent 471081353216
item 62 key (470810787840 169 0) itemoff 12674 itemsize
33
extent refs 1 gen 4519492 flags 2
ref#0: tree block backref root 398
item 63 key (470810804224 169 0) itemoff 12641 itemsize
33
extent refs 1 gen 4814304 flags 258
ref#0: tree block backref root 480469
item 64 key (470811000832 169 0) itemoff 12608 itemsize
33
extent refs 1 gen 4865321 flags 258
ref#0: shared block backref parent 736748601344
item 65 key (470811049984 169 0) itemoff 12575 itemsize
33
extent refs 1 gen 4833910 flags 258
ref#0: shared block backref parent 471055040512
item 66 key (470811181056 169 0) itemoff 12524 itemsize
51
extent refs 3 gen 4838387 flags 2
ref#0: tree block backref root 398
ref#1: shared block backref parent 737420591104
ref#2: shared block backref parent 350765498368
item 67 key (470811197440 169 0) itemoff 12491 itemsize
33
extent refs 1 gen 4864783 flags 2
ref#0: tree block backref root 2
item 68 key (470811213824 169 1) itemoff 12242 itemsize
249
extent refs 25 gen 4541732 flags 2
ref#0: tree block backref root 486052
ref#1: tree block backref root 486051
ref#2: tree block backref root 486050
ref#3: tree block backref root 486049
ref#4: tree block backref root 486048
ref#5: tree block backref root 486047
ref#6: tree block backref root 486046
ref#7: tree block backref root 486045
ref#8: tree block backref root 486044
ref#9: tree block backref root 486043
ref#10: tree block backref root 486035
ref#11: tree block backref root 486031
ref#12: tree block backref root 486018
ref#13: tree block backref root 486005
ref#14: tree block backref root 485992
ref#15: tree block backref root 485979
ref#16: tree block backref root 485966
ref#17: tree block backref root 485953
ref#18: tree block backref root 485721
ref#19: tree block backref root 485408
ref#20: tree block backref root 484779
ref#21: tree block backref root 482587
ref#22: tree block backref root 480469
ref#23: tree block backref root 478601
ref#24: tree block backref root 398
item 69 key (470811230208 169 0) itemoff 12209 itemsize
33
extent refs 1 gen 4541732 flags 2
ref#0: tree block backref root 398
item 70 key (470811262976 169 1) itemoff 11960 itemsize
249
extent refs 25 gen 4541732 flags 2
ref#0: tree block backref root 486052
ref#1: tree block backref root 486051
ref#2: tree block backref root 486050
ref#3: tree block backref root 486049
ref#4: tree block backref root 486048
ref#5: tree block backref root 486047
ref#6: tree block backref root 486046
ref#7: tree block backref root 486045
ref#8: tree block backref root 486044
ref#9: tree block backref root 486043
ref#10: tree block backref root 486035
ref#11: tree block backref root 486031
ref#12: tree block backref root 486018
ref#13: tree block backref root 486005
ref#14: tree block backref root 485992
ref#15: tree block backref root 485979
ref#16: tree block backref root 485966
ref#17: tree block backref root 485953
ref#18: tree block backref root 485721
ref#19: tree block backref root 485408
ref#20: tree block backref root 484779
ref#21: tree block backref root 482587
ref#22: tree block backref root 480469
ref#23: tree block backref root 478601
ref#24: tree block backref root 398
item 71 key (470811279360 169 0) itemoff 11927 itemsize
33
extent refs 1 gen 4541732 flags 2
ref#0: tree block backref root 398
item 72 key (470811344896 169 0) itemoff 11894 itemsize
33
extent refs 1 gen 4541732 flags 2
ref#0: tree block backref root 398
item 73 key (470811361280 169 0) itemoff 11843 itemsize
51
extent refs 3 gen 4840882 flags 258
ref#0: tree block backref root 485721
ref#1: tree block backref root 485408
ref#2: tree block backref root 484779
item 74 key (470811377664 169 0) itemoff 11648 itemsize
195
extent refs 19 gen 4671752 flags 2
ref#0: tree block backref root 398
ref#1: shared block backref parent 737641398272
ref#2: shared block backref parent 737490583552
ref#3: shared block backref parent 737216856064
ref#4: shared block backref parent 737046052864
ref#5: shared block backref parent 736820871168
ref#6: shared block backref parent 736769540096
ref#7: shared block backref parent 471276994560
ref#8: shared block backref parent 470476406784
ref#9: shared block backref parent 350562942976
ref#10: shared block backref parent 350443454464
ref#11: shared block backref parent 350179344384
ref#12: shared block backref parent 349865132032
ref#13: shared block backref parent 349797400576
ref#14: shared block backref parent 349431463936
ref#15: shared block backref parent 153274597376
ref#16: shared block backref parent 152757616640
ref#17: shared block backref parent 51273973760
ref#18: shared block backref parent 50575310848
item 75 key (470811426816 169 0) itemoff 11570 itemsize
78
extent refs 6 gen 4838387 flags 2
ref#0: tree block backref root 398
ref#1: shared block backref parent 470774349824
ref#2: shared block backref parent 350775836672
ref#3: shared block backref parent 51398656000
ref#4: shared block backref parent 51369590784
ref#5: shared block backref parent 759103488
item 76 key (470811443200 169 1) itemoff 11321 itemsize
249
extent refs 25 gen 4541732 flags 2
ref#0: tree block backref root 486052
ref#1: tree block backref root 486051
ref#2: tree block backref root 486050
ref#3: tree block backref root 486049
ref#4: tree block backref root 486048
ref#5: tree block backref root 486047
ref#6: tree block backref root 486046
ref#7: tree block backref root 486045
ref#8: tree block backref root 486044
ref#9: tree block backref root 486043
ref#10: tree block backref root 486035
ref#11: tree block backref root 486031
ref#12: tree block backref root 486018
ref#13: tree block backref root 486005
ref#14: tree block backref root 485992
ref#15: tree block backref root 485979
ref#16: tree block backref root 485966
ref#17: tree block backref root 485953
ref#18: tree block backref root 485721
ref#19: tree block backref root 485408
ref#20: tree block backref root 484779
ref#21: tree block backref root 482587
ref#22: tree block backref root 480469
ref#23: tree block backref root 478601
ref#24: tree block backref root 398
item 77 key (470811475968 169 0) itemoff 11288 itemsize
33
extent refs 1 gen 4830831 flags 2
ref#0: tree block backref root 2
item 78 key (470811492352 169 0) itemoff 11255 itemsize
33
extent refs 1 gen 4541732 flags 2
ref#0: tree block backref root 398
item 79 key (470811508736 169 0) itemoff 11213 itemsize
42
extent refs 2 gen 3042608 flags 2
ref#0: tree block backref root 398
ref#1: shared block backref parent 349646520320
item 80 key (470811525120 169 0) itemoff 11180 itemsize
33
extent refs 1 gen 3577077 flags 2
ref#0: tree block backref root 398
item 81 key (470811541504 169 1) itemoff 10931 itemsize
249
extent refs 25 gen 4541732 flags 2
ref#0: tree block backref root 486052
ref#1: tree block backref root 486051
ref#2: tree block backref root 486050
ref#3: tree block backref root 486049
ref#4: tree block backref root 486048
ref#5: tree block backref root 486047
ref#6: tree block backref root 486046
ref#7: tree block backref root 486045
ref#8: tree block backref root 486044
ref#9: tree block backref root 486043
ref#10: tree block backref root 486035
ref#11: tree block backref root 486031
ref#12: tree block backref root 486018
ref#13: tree block backref root 486005
ref#14: tree block backref root 485992
ref#15: tree block backref root 485979
ref#16: tree block backref root 485966
ref#17: tree block backref root 485953
ref#18: tree block backref root 485721
ref#19: tree block backref root 485408
ref#20: tree block backref root 484779
ref#21: tree block backref root 482587
ref#22: tree block backref root 480469
ref#23: tree block backref root 478601
ref#24: tree block backref root 398
item 82 key (470811557888 169 0) itemoff 10898 itemsize
33
extent refs 1 gen 4541732 flags 2
ref#0: tree block backref root 398
item 83 key (470811590656 169 0) itemoff 10865 itemsize
33
extent refs 1 gen 4541732 flags 2
ref#0: tree block backref root 398
item 84 key (470811639808 169 0) itemoff 10832 itemsize
33
extent refs 1 gen 4541732 flags 2
ref#0: tree block backref root 398
item 85 key (470811656192 169 0) itemoff 10799 itemsize
33
extent refs 1 gen 3547360 flags 2
ref#0: tree block backref root 398
item 86 key (470811672576 169 0) itemoff 10766 itemsize
33
extent refs 1 gen 3280274 flags 2
ref#0: tree block backref root 398
item 87 key (470811688960 169 0) itemoff 10733 itemsize
33
extent refs 1 gen 3280274 flags 2
ref#0: tree block backref root 398
item 88 key (470811738112 169 0) itemoff 10700 itemsize
33
extent refs 1 gen 4541732 flags 2
ref#0: tree block backref root 398
item 89 key (470811770880 169 0) itemoff 10667 itemsize
33
extent refs 1 gen 4541732 flags 2
ref#0: tree block backref root 398
item 90 key (470811803648 169 0) itemoff 10634 itemsize
33
extent refs 1 gen 4813689 flags 258
ref#0: shared block backref parent 471114399744
item 91 key (470811836416 169 1) itemoff 10592 itemsize
42
extent refs 2 gen 4541732 flags 258
ref#0: tree block backref root 480469
ref#1: tree block backref root 478601
item 92 key (470811869184 169 0) itemoff 10550 itemsize
42
extent refs 2 gen 4541732 flags 2
ref#0: tree block backref root 398
ref#1: shared block backref parent 470811836416
item 93 key (470811901952 169 0) itemoff 10355 itemsize
195
extent refs 19 gen 4671752 flags 2
ref#0: tree block backref root 398
ref#1: shared block backref parent 737641398272
ref#2: shared block backref parent 737490583552
ref#3: shared block backref parent 737216856064
ref#4: shared block backref parent 737046052864
ref#5: shared block backref parent 736820871168
ref#6: shared block backref parent 736769540096
ref#7: shared block backref parent 471276994560
ref#8: shared block backref parent 470476406784
ref#9: shared block backref parent 350562942976
ref#10: shared block backref parent 350443454464
ref#11: shared block backref parent 350179344384
ref#12: shared block backref parent 349865132032
ref#13: shared block backref parent 349797400576
ref#14: shared block backref parent 349431463936
ref#15: shared block backref parent 153274597376
ref#16: shared block backref parent 152757616640
ref#17: shared block backref parent 51273973760
ref#18: shared block backref parent 50575310848
item 94 key (470811967488 169 0) itemoff 10295 itemsize
60
extent refs 4 gen 4807178 flags 2
ref#0: tree block backref root 398
ref#1: shared block backref parent 736921075712
ref#2: shared block backref parent 351076073472
ref#3: shared block backref parent 51439304704
item 95 key (470811983872 169 0) itemoff 10262 itemsize
33
extent refs 1 gen 4864449 flags 2
ref#0: tree block backref root 2
item 96 key (470812065792 169 0) itemoff 10229 itemsize
33
extent refs 1 gen 4864449 flags 2
ref#0: tree block backref root 2
item 97 key (470812098560 169 0) itemoff 10196 itemsize
33
extent refs 1 gen 3786399 flags 2
ref#0: tree block backref root 372498
item 98 key (470812147712 169 0) itemoff 10163 itemsize
33
extent refs 1 gen 4813689 flags 258
ref#0: shared block backref parent 471114399744
item 99 key (470812196864 169 0) itemoff 10103 itemsize
60
extent refs 4 gen 4807178 flags 2
ref#0: tree block backref root 398
ref#1: shared block backref parent 736921075712
ref#2: shared block backref parent 351076073472
ref#3: shared block backref parent 51439304704
item 100 key (470812327936 169 0) itemoff 10070 itemsize
33
extent refs 1 gen 4813689 flags 258
ref#0: shared block backref parent 351087575040
item 101 key (470812393472 169 0) itemoff 10010 itemsize
60
extent refs 4 gen 4807178 flags 2
ref#0: tree block backref root 398
ref#1: shared block backref parent 736921075712
ref#2: shared block backref parent 351076073472
ref#3: shared block backref parent 51439304704
item 102 key (470812426240 169 0) itemoff 9950 itemsize
60
extent refs 4 gen 4813689 flags 2
ref#0: tree block backref root 398
ref#1: shared block backref parent 351087575040
ref#2: shared block backref parent 51350798336
ref#3: shared block backref parent 50696880128
item 103 key (470812442624 169 0) itemoff 9890 itemsize
60
extent refs 4 gen 4807178 flags 2
ref#0: tree block backref root 398
ref#1: shared block backref parent 736921075712
ref#2: shared block backref parent 351076073472
ref#3: shared block backref parent 51439304704
item 104 key (470812508160 169 0) itemoff 9857 itemsize
33
extent refs 1 gen 4795277 flags 258
ref#0: shared block backref parent 152638423040
item 105 key (470812524544 169 0) itemoff 9797 itemsize
60
extent refs 4 gen 4807178 flags 2
ref#0: tree block backref root 398
ref#1: shared block backref parent 736921075712
ref#2: shared block backref parent 351076073472
ref#3: shared block backref parent 51439304704
item 106 key (470812639232 169 0) itemoff 9683 itemsize
114
extent refs 10 gen 4808224 flags 2
ref#0: tree block backref root 398
ref#1: shared block backref parent 737597767680
ref#2: shared block backref parent 736663928832
ref#3: shared block backref parent 736625934336
ref#4: shared block backref parent 471068426240
ref#5: shared block backref parent 470548611072
ref#6: shared block backref parent 350823383040
ref#7: shared block backref parent 349530652672
ref#8: shared block backref parent 51197263872
ref#9: shared block backref parent 50881314816
item 107 key (470812655616 169 0) itemoff 9632 itemsize
51
extent refs 3 gen 4865700 flags 2
ref#0: tree block backref root 398
ref#1: shared block backref parent 737392377856
ref#2: shared block backref parent 471391862784
item 108 key (470812688384 169 0) itemoff 9581 itemsize
51
extent refs 3 gen 4865700 flags 2
ref#0: tree block backref root 398
ref#1: shared block backref parent 737392771072
ref#2: shared block backref parent 471391895552
item 109 key (470812721152 169 0) itemoff 9548 itemsize
33
extent refs 1 gen 4796010 flags 258
ref#0: shared block backref parent 152641617920
item 110 key (470812753920 169 0) itemoff 9515 itemsize
33
extent refs 1 gen 4833020 flags 258
ref#0: shared block backref parent 471057973248
item 111 key (470812786688 169 0) itemoff 9482 itemsize
33
extent refs 1 gen 4833020 flags 258
ref#0: shared block backref parent 471057973248
item 112 key (470812835840 169 0) itemoff 9449 itemsize
33
extent refs 1 gen 4814184 flags 258
ref#0: tree block backref root 480469
item 113 key (470812868608 169 0) itemoff 9416 itemsize
33
extent refs 1 gen 4814184 flags 258
ref#0: shared block backref parent 152649089024
item 114 key (470812901376 169 0) itemoff 9383 itemsize
33
extent refs 1 gen 3786399 flags 2
ref#0: tree block backref root 372498
item 115 key (470812917760 169 0) itemoff 9350 itemsize
33
extent refs 1 gen 3786399 flags 2
ref#0: tree block backref root 372498
item 116 key (470812934144 169 0) itemoff 9317 itemsize
33
extent refs 1 gen 4814184 flags 258
ref#0: shared block backref parent 736612941824
item 117 key (470812999680 169 0) itemoff 9284 itemsize
33
extent refs 1 gen 3192658 flags 2
ref#0: tree block backref root 398
item 118 key (470813065216 169 0) itemoff 9233 itemsize
51
extent refs 3 gen 4865321 flags 258
ref#0: shared block backref parent 736775798784
ref#1: shared block backref parent 736755400704
ref#2: shared block backref parent 349566222336
item 119 key (470813130752 169 0) itemoff 9200 itemsize
33
extent refs 1 gen 4813730 flags 258
ref#0: shared block backref parent 152772853760
item 120 key (470813163520 169 0) itemoff 9158 itemsize
42
extent refs 2 gen 3753422 flags 2
ref#0: tree block backref root 398
ref#1: shared block backref parent 51333234688
item 121 key (470813212672 169 0) itemoff 9125 itemsize
33
extent refs 1 gen 3582623 flags 2
ref#0: tree block backref root 398
item 122 key (470813261824 169 0) itemoff 9002 itemsize
123
extent refs 11 gen 4838387 flags 2
ref#0: tree block backref root 398
ref#1: shared block backref parent 737545355264
ref#2: shared block backref parent 737225555968
ref#3: shared block backref parent 470698000384
ref#4: shared block backref parent 470578429952
ref#5: shared block backref parent 350471487488
ref#6: shared block backref parent 153275826176
ref#7: shared block backref parent 152577064960
ref#8: shared block backref parent 51433160704
ref#9: shared block backref parent 51368902656
ref#10: shared block backref parent 1028685824
item 123 key (470813327360 169 0) itemoff 8969 itemsize
33
extent refs 1 gen 4848605 flags 258
ref#0: shared block backref parent 736993067008
item 124 key (470813343744 169 0) itemoff 8936 itemsize
33
extent refs 1 gen 3582623 flags 2
ref#0: tree block backref root 398
item 125 key (470813376512 169 0) itemoff 8903 itemsize
33
extent refs 1 gen 4813730 flags 258
ref#0: shared block backref parent 152773083136
item 126 key (470813392896 169 0) itemoff 8825 itemsize
78
extent refs 6 gen 4838387 flags 2
ref#0: tree block backref root 398
ref#1: shared block backref parent 470774349824
ref#2: shared block backref parent 350775836672
ref#3: shared block backref parent 51398656000
ref#4: shared block backref parent 51369590784
ref#5: shared block backref parent 759103488
item 127 key (470813425664 169 0) itemoff 8774 itemsize
51
extent refs 3 gen 4838387 flags 2
ref#0: tree block backref root 398
ref#1: shared block backref parent 737420591104
ref#2: shared block backref parent 350765498368
item 128 key (470813491200 169 0) itemoff 8732 itemsize
42
extent refs 2 gen 4865525 flags 258
ref#0: shared block backref parent 736771162112
ref#1: shared block backref parent 349568221184
item 129 key (470813507584 169 0) itemoff 8690 itemsize
42
extent refs 2 gen 4865525 flags 258
ref#0: shared block backref parent 736768950272
ref#1: shared block backref parent 349568073728
item 130 key (470813622272 169 0) itemoff 8639 itemsize
51
extent refs 3 gen 4838387 flags 2
ref#0: tree block backref root 398
ref#1: shared block backref parent 737420591104
ref#2: shared block backref parent 350765498368
item 131 key (470813655040 169 0) itemoff 8606 itemsize
33
extent refs 1 gen 4813730 flags 258
ref#0: shared block backref parent 152773083136
item 132 key (470813671424 169 0) itemoff 8555 itemsize
51
extent refs 3 gen 4865700 flags 2
ref#0: tree block backref root 398
ref#1: shared block backref parent 737392771072
ref#2: shared block backref parent 471391895552
item 133 key (470813687808 169 0) itemoff 8522 itemsize
33
extent refs 1 gen 4865525 flags 2
ref#0: tree block backref root 398
item 134 key (470813704192 169 0) itemoff 8489 itemsize
33
extent refs 1 gen 4865525 flags 258
ref#0: shared block backref parent 470777266176
item 135 key (470813720576 169 0) itemoff 8438 itemsize
51
extent refs 3 gen 4865700 flags 2
ref#0: tree block backref root 398
ref#1: shared block backref parent 737392377856
ref#2: shared block backref parent 471391862784
item 136 key (470813736960 169 0) itemoff 8405 itemsize
33
extent refs 1 gen 3275257 flags 2
ref#0: tree block backref root 398
item 137 key (470813753344 169 0) itemoff 8372 itemsize
33
extent refs 1 gen 3717914 flags 2
ref#0: tree block backref root 398
item 138 key (470813786112 169 0) itemoff 8321 itemsize
51
extent refs 3 gen 4865700 flags 2
ref#0: tree block backref root 398
ref#1: shared block backref parent 737392771072
ref#2: shared block backref parent 471391895552
item 139 key (470813884416 169 0) itemoff 8288 itemsize
33
extent refs 1 gen 4865067 flags 258
ref#0: shared block backref parent 737247936512
item 140 key (470813917184 169 0) itemoff 8237 itemsize
51
extent refs 3 gen 4864776 flags 2
ref#0: tree block backref root 398
ref#1: shared block backref parent 350761287680
ref#2: shared block backref parent 51255246848
item 141 key (470813999104 169 0) itemoff 8141 itemsize
96
extent refs 8 gen 4664339 flags 2
ref#0: tree block backref root 398
ref#1: shared block backref parent 737305460736
ref#2: shared block backref parent 736612745216
ref#3: shared block backref parent 471363321856
ref#4: shared block backref parent 350379294720
ref#5: shared block backref parent 349652680704
ref#6: shared block backref parent 349115236352
ref#7: shared block backref parent 51438632960
item 142 key (470814015488 169 0) itemoff 8108 itemsize
33
extent refs 1 gen 4808224 flags 258
ref#0: shared block backref parent 471068426240
item 143 key (470814031872 169 0) itemoff 8075 itemsize
33
extent refs 1 gen 4864783 flags 2
ref#0: tree block backref root 2
item 144 key (470814048256 169 0) itemoff 8042 itemsize
33
extent refs 1 gen 3446703 flags 2
ref#0: tree block backref root 398
item 145 key (470814064640 169 0) itemoff 8009 itemsize
33
extent refs 1 gen 2863244 flags 2
ref#0: tree block backref root 398
item 146 key (470814081024 169 0) itemoff 7976 itemsize
33
extent refs 1 gen 3446703 flags 2
ref#0: tree block backref root 398
item 147 key (470814212096 169 0) itemoff 7943 itemsize
33
extent refs 1 gen 3268842 flags 2
ref#0: tree block backref root 398
item 148 key (470814277632 169 0) itemoff 7847 itemsize
96
extent refs 8 gen 4664339 flags 2
ref#0: tree block backref root 398
ref#1: shared block backref parent 737305460736
ref#2: shared block backref parent 736612745216
ref#3: shared block backref parent 471363321856
ref#4: shared block backref parent 350379294720
ref#5: shared block backref parent 349652680704
ref#6: shared block backref parent 349115236352
ref#7: shared block backref parent 51438632960
item 149 key (470814375936 169 0) itemoff 7751 itemsize
96
extent refs 8 gen 4679244 flags 2
ref#0: tree block backref root 398
ref#1: shared block backref parent 737305460736
ref#2: shared block backref parent 736612745216
ref#3: shared block backref parent 471363321856
ref#4: shared block backref parent 350379294720
ref#5: shared block backref parent 349652680704
ref#6: shared block backref parent 349115236352
ref#7: shared block backref parent 51438632960
item 150 key (470814408704 169 0) itemoff 7511 itemsize
240
extent refs 24 gen 4803092 flags 2
ref#0: tree block backref root 398
ref#1: shared block backref parent 737481834496
ref#2: shared block backref parent 737393131520
ref#3: shared block backref parent 737301790720
ref#4: shared block backref parent 737200062464
ref#5: shared block backref parent 736778551296
ref#6: shared block backref parent 736749125632
ref#7: shared block backref parent 736747782144
ref#8: shared block backref parent 736705806336
ref#9: shared block backref parent 471389224960
ref#10: shared block backref parent 471052976128
ref#11: shared block backref parent 470802137088
ref#12: shared block backref parent 470767534080
ref#13: shared block backref parent 350429577216
ref#14: shared block backref parent 350375116800
ref#15: shared block backref parent 349566599168
ref#16: shared block backref parent 349356081152
ref#17: shared block backref parent 153246482432
ref#18: shared block backref parent 153021202432
ref#19: shared block backref parent 152997888000
ref#20: shared block backref parent 152630607872
ref#21: shared block backref parent 51421954048
ref#22: shared block backref parent 51236110336
ref#23: shared block backref parent 50828017664
item 151 key (470814441472 169 0) itemoff 7289 itemsize 222
extent refs 22 gen 4838387 flags 2
ref#0: tree block backref root 486052
ref#1: tree block backref root 486051
ref#2: tree block backref root 486050
ref#3: tree block backref root 486049
ref#4: tree block backref root 486048
ref#5: tree block backref root 486047
ref#6: tree block backref root 486046
ref#7: tree block backref root 486045
ref#8: tree block backref root 486044
ref#9: tree block backref root 486043
ref#10: tree block backref root 486035
ref#11: tree block backref root 486031
ref#12: tree block backref root 486018
ref#13: tree block backref root 486005
ref#14: tree block backref root 485992
ref#15: tree block backref root 485979
ref#16: tree block backref root 485966
ref#17: tree block backref root 485953
ref#18: tree block backref root 485721
ref#19: tree block backref root 485408
ref#20: tree block backref root 484779
ref#21: tree block backref root 398
item 152 key (470814457856 169 0) itemoff 7256 itemsize 33
extent refs 1 gen 4838485 flags 2
ref#0: tree block backref root 7
item 153 key (470814507008 169 0) itemoff 7223 itemsize 33
extent refs 1 gen 4864783 flags 2
ref#0: tree block backref root 2
item 154 key (470814539776 169 0) itemoff 7190 itemsize 33
extent refs 1 gen 4830938 flags 258
ref#0: shared block backref parent 737199390720
item 155 key (470814556160 169 0) itemoff 7157 itemsize 33
extent refs 1 gen 4864783 flags 2
ref#0: tree block backref root 2
item 156 key (470814588928 169 0) itemoff 7124 itemsize 33
extent refs 1 gen 3658861 flags 2
ref#0: tree block backref root 398
item 157 key (470814621696 169 0) itemoff 7091 itemsize 33
extent refs 1 gen 3295735 flags 2
ref#0: tree block backref root 398
item 158 key (470814638080 169 0) itemoff 7058 itemsize 33
extent refs 1 gen 4864783 flags 2
ref#0: tree block backref root 2
item 159 key (470814654464 169 0) itemoff 7025 itemsize 33
extent refs 1 gen 4864783 flags 2
ref#0: tree block backref root 2
item 160 key (470814703616 169 0) itemoff 6992 itemsize 33
extent refs 1 gen 3268842 flags 2
ref#0: tree block backref root 398
item 161 key (470814736384 169 1) itemoff 6959 itemsize 33
extent refs 1 gen 4857403 flags 258
ref#0: shared block backref parent 470478274560
item 162 key (470814752768 169 0) itemoff 6926 itemsize 33
extent refs 1 gen 3658861 flags 2
ref#0: tree block backref root 398
item 163 key (470814834688 169 0) itemoff 6893 itemsize 33
extent refs 1 gen 3786399 flags 2
ref#0: tree block backref root 372498
item 164 key (470814916608 169 0) itemoff 6833 itemsize 60
extent refs 4 gen 3935308 flags 2
ref#0: tree block backref root 398
ref#1: shared block backref parent 349894819840
ref#2: shared block backref parent 50765283328
ref#3: shared block backref parent 924663808
item 165 key (470814949376 169 0) itemoff 6800 itemsize 33
extent refs 1 gen 4830938 flags 258
ref#0: shared block backref parent 152888557568
item 166 key (470815031296 169 0) itemoff 6659 itemsize 141
extent refs 13 gen 4863749 flags 2
ref#0: tree block backref root 398
ref#1: shared block backref parent 737641398272
ref#2: shared block backref parent 737216856064
ref#3: shared block backref parent 737046052864
ref#4: shared block backref parent 736820871168
ref#5: shared block backref parent 736769540096
ref#6: shared block backref parent 471276994560
ref#7: shared block backref parent 350443454464
ref#8: shared block backref parent 350179344384
ref#9: shared block backref parent 349797400576
ref#10: shared block backref parent 153274597376
ref#11: shared block backref parent 152757616640
ref#12: shared block backref parent 50575310848
item 167 key (470815145984 169 0) itemoff 6599 itemsize 60
extent refs 4 gen 4807178 flags 2
ref#0: tree block backref root 398
ref#1: shared block backref parent 736921075712
ref#2: shared block backref parent 351076073472
ref#3: shared block backref parent 51439304704
item 168 key (470815178752 169 0) itemoff 6566 itemsize 33
extent refs 1 gen 3295735 flags 2
ref#0: tree block backref root 398
BTRFS critical (device dm-4): adding refs to an existing tree ref, bytenr 470809264128 num_bytes 16384 root_objectid 398 slot 1
BTRFS error (device dm-4): failed to run delayed ref for logical 470809264128 num_bytes 16384 type 176 action 1 ref_mod 1: -117
------------[ cut here ]------------
BTRFS: Transaction aborted (error -117)
WARNING: CPU: 4 PID: 2518421 at fs/btrfs/extent-tree.c:2199 btrfs_run_delayed_refs+0x107/0x140
Modules linked in: hfs cdrom msdos jfs nls_ucs2_utils cdc_acm ch341 usbserial exfat sg uas usb_storage rpcsec_gss_krb5 nfsv4 dns_resolver nfs netfs nf_conntrack_netlink xfrm_user xfrm_algo xt_addrtype br_netfilter bridge stp llc xt_tcpudp xt_conntrack rfcomm snd_seq_dummy snd_hrtimer ccm overlay ipt_REJECT nf_reject_ipv4 xt_MASQUERADE xt_LOG nf_log_syslog nft_compat nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 cmac algif_hash nf_tables algif_skcipher af_alg bnep binfmt_misc btusb btrtl uvcvideo btbcm videobuf2_vmalloc btmtk uvc btintel videobuf2_memops videobuf2_v4l2 videodev bluetooth videobuf2_common mc nls_utf8 nls_cp437 vfat fat squashfs loop iwlmvm intel_uncore_frequency intel_uncore_frequency_common intel_tcc_cooling mac80211 snd_hda_codec_hdmi x86_pkg_temp_thermal intel_powerclamp libarc4 snd_hda_codec_realtek kvm_intel snd_hda_codec_generic snd_soc_dmic snd_hda_scodec_component iwlwifi snd_sof_pci_intel_tgl snd_sof_pci_intel_cnl snd_sof_intel_hda_generic kvm
snd_sof_intel_hda_common snd_sof_intel_hda snd_sof_pci processor_thermal_device_pci_legacy snd_sof_xtensa_dsp processor_thermal_device snd_soc_hdac_hda processor_thermal_wt_hint snd_soc_acpi_intel_match processor_thermal_rfim snd_soc_acpi mei_pxp mei_hdcp thinkpad_acpi tpm_crb processor_thermal_rapl rapl nvram cfg80211 snd_soc_avs intel_cstate platform_profile intel_rapl_common nvidiafb pcspkr ucsi_acpi snd_soc_hda_codec processor_thermal_wt_req think_lmi iTCO_wdt vgastate processor_thermal_power_floor firmware_attributes_class typec_ucsi wmi_bmof fb_ddc ee1004 iTCO_vendor_support snd_hda_intel typec mei_me processor_thermal_mbox intel_soc_dts_iosf roles int3403_thermal rfkill ac int340x_thermal_zone intel_pmc_core intel_vsec tpm_tis pmt_telemetry int3400_thermal intel_hid tpm_tis_core pmt_class acpi_pad acpi_tad acpi_thermal_rel sparse_keymap input_leds joydev evdev serio_raw vboxdrv(OE) soundwire_intel soundwire_cadence snd_sof_intel_hda_mlink soundwire_generic_allocation snd_sof_probes snd_sof
snd_sof_utils snd_intel_dspcfg snd_intel_sdw_acpi snd_soc_skl_hda_dsp snd_soc_intel_hda_dsp_common snd_hda_codec snd_hwdep snd_soc_hdac_hdmi snd_hda_ext_core snd_soc_core snd_compress snd_pcm_dmaengine snd_hda_core snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_midi snd_seq_midi_event snd_seq snd_timer snd_rawmidi snd_seq_device snd_ctl_led snd soundcore ac97_bus configs coretemp msr fuse efi_pstore nfsd auth_rpcgss nfs_acl lockd grace sunrpc nfnetlink ip_tables x_tables autofs4 essiv authenc dm_crypt trusted asn1_encoder tee tpm rng_core libaescfb ecdh_generic dm_mod ecc raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx sata_sil24 r8169 realtek mdio_devres libphy mii hid_generic usbhid hid i915 drm_buddy xhci_pci i2c_algo_bit ttm crct10dif_pclmul xhci_hcd crc32_pclmul drm_display_helper crc32c_intel rtsx_pci_sdmmc polyval_clmulni cec polyval_generic mmc_core usbcore rc_core i2c_i801 video ptp i2c_mux ghash_clmulni_intel spi_intel_pci sha512_ssse3 sha256_ssse3 psmouse sha1_ssse3 rtsx_pci
thunderbolt spi_intel i2c_smbus pps_core hwmon usb_common thermal battery wmi aesni_intel crypto_simd cryptd [last unloaded: igc]
CPU: 4 UID: 0 PID: 2518421 Comm: btrfs Tainted: G U OE 6.11.2-amd64-preempt-sysrq-20241007 #1 1a512c2db5f087f236d90ecfb30551fddcc51243
Tainted: [U]=USER, [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
Hardware name: LENOVO 20YU002JUS/20YU002JUS, BIOS N37ET49W (1.30 ) 11/15/2023
RIP: 0010:btrfs_run_delayed_refs+0x107/0x140
Code: 01 00 00 00 eb b6 e8 18 8e b7 00 31 db 89 d8 5b 5d 41 5c 41 5d 41 5e c3 cc cc cc cc 89 de 48 c7 c7 40 bb b7 93 e8 f9 0c 9f ff <0f> 0b eb d0 66 66 2e 0f 1f 84 00 00 00 00 00 66 66 2e 0f 1f 84 00
RSP: 0018:ffffade4eb1bb728 EFLAGS: 00010282
RAX: 0000000000000000 RBX: 00000000ffffff8b RCX: 0000000000000027
RDX: ffff92922f421848 RSI: 0000000000000001 RDI: ffff92922f421840
RBP: ffff9273b9581930 R08: 0000000000000000 R09: 0000000000000003
R10: ffffade4eb1bb5c8 R11: ffff9292af7d5028 R12: 0000000000000000
R13: ffff9278edc36358 R14: ffff9278edc36200 R15: 0000000000000000
FS: 00007fbb83c6a380(0000) GS:ffff92922f400000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00005609e129c078 CR3: 000000137eeb6002 CR4: 0000000000770ef0
PKRU: 55555554
Call Trace:
<TASK>
? __warn+0x7c/0x140
? btrfs_run_delayed_refs+0x107/0x140
? report_bug+0x160/0x1c0
? handle_bug+0x41/0x80
? exc_invalid_op+0x15/0x100
? asm_exc_invalid_op+0x16/0x40
? btrfs_run_delayed_refs+0x107/0x140
? btrfs_run_delayed_refs+0x107/0x140
btrfs_commit_transaction+0x69/0xe80
? btrfs_update_reloc_root+0x12d/0x240
prepare_to_merge+0x4f0/0x600
relocate_block_group+0x113/0x500
btrfs_relocate_block_group+0x27a/0x440
btrfs_relocate_chunk+0x3b/0x180
btrfs_balance+0x8c1/0x1340
? btrfs_ioctl+0x18db/0x26c0
btrfs_ioctl+0x2285/0x26c0
? vm_area_alloc+0x59/0xc0
? chacha_block_generic+0x6f/0xc0
? __rmqueue_pcplist+0xa9/0xd80
? __rmqueue_pcplist+0xa9/0xd80
__x64_sys_ioctl+0x90/0x100
do_syscall_64+0x69/0x140
? get_page_from_freelist+0x5ab/0x1900
? __alloc_pages_noprof+0x17f/0x380
? __mod_memcg_lruvec_state+0x91/0x140
? __lruvec_stat_mod_folio+0x7f/0x100
? __pte_offset_map_lock+0x92/0x140
? set_ptes.constprop.0+0x41/0xc0
? do_anonymous_page+0xfa/0x800
? __pte_offset_map+0x17/0x180
? __handle_mm_fault+0x852/0x880
? __count_memcg_events+0x54/0x100
? count_memcg_events.constprop.0+0x1a/0x40
? handle_mm_fault+0xaa/0x300
? do_user_addr_fault+0x375/0x680
? clear_bhb_loop+0x45/0xc0
? clear_bhb_loop+0x45/0xc0
? clear_bhb_loop+0x45/0xc0
? clear_bhb_loop+0x45/0xc0
? clear_bhb_loop+0x45/0xc0
entry_SYSCALL_64_after_hwframe+0x76/0x7e
RIP: 0033:0x7fbb83d834bb
Code: 00 48 89 44 24 18 31 c0 48 8d 44 24 60 c7 04 24 10 00 00 00 48 89 44 24 08 48 8d 44 24 20 48 89 44 24 10 b8 10 00 00 00 0f 05 <89> c2 3d 00 f0 ff ff 77 1c 48 8b 44 24 18 64 48 2b 04 25 28 00 00
RSP: 002b:00007ffc4097f6b0 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007fbb83d834bb
RDX: 00007ffc4097f718 RSI: 00000000c4009420 RDI: 0000000000000003
RBP: 0000000000000003 R08: 00007fbb83e5eac0 R09: 0000000000000004
R10: 0000000000000000 R11: 0000000000000246 R12: 00007fbb83e5e248
R13: 00007ffc40981e98 R14: 000055c3c393515c R15: 0000000000000001
</TASK>
---[ end trace 0000000000000000 ]---
BTRFS: error (device dm-4 state A) in btrfs_run_delayed_refs:2199: errno=-117 Filesystem corrupted
BTRFS info (device dm-4 state EA): forced readonly
BTRFS info (device dm-4 state EA): balance: ended with status: -30
--
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: BTRFS error (device dm-4): failed to run delayed ref for logical 350223581184 num_bytes 16384 type 176 action 1 ref_mod 1: -117 (kernel 6.11.2)
2025-02-27 14:11 ` Marc MERLIN
@ 2025-02-27 15:05 ` Marc MERLIN
2025-02-28 13:10 ` Marc MERLIN
1 sibling, 0 replies; 17+ messages in thread
From: Marc MERLIN @ 2025-02-27 15:05 UTC (permalink / raw)
To: Qu Wenruo
Cc: Qu Wenruo, linux-btrfs, Josef Bacik, Chris Murphy, Zygo Blaxell,
Roman Mamedov, Su Yue
[-- Attachment #1: Type: text/plain, Size: 2905 bytes --]
On Thu, Feb 27, 2025 at 06:11:16AM -0800, Marc MERLIN wrote:
> On Tue, Feb 25, 2025 at 07:52:11PM +1030, Qu Wenruo wrote:
> >
> >
> > 在 2025/2/25 19:48, Marc MERLIN 写道:
> > > On Tue, Feb 25, 2025 at 07:16:31PM +1030, Qu Wenruo wrote:
> > > > Ref#13 is the root that mentioned has the existing one.
> > > >
> > > > I have no idea why it shows up like this.
> > > >
> > > > But since the fs passes btrfs check, mind to dump the following tree block?
> > > >
> > > > # btrfs ins dump-tree -t extent <device> | grep "(350223581184 " -A 50
> > > >
> > > > I want to make sure if the ref 398 exists on disk, or it's generated at
> > > > runtime.
> > >
> > > sauron:~# btrfs ins dump-tree -t extent /dev/mapper/pool1 | grep "350223581184 " -A 50
> > > sauron:~#
> > >
> > > Mmmmh, so 350223581184 is gone now since it's been 20 days already since
> > > my original post.
> > > I can try to re-enable balance and see if things crash or not.
>
> Ok, I didn't run memtest last night, but ran a new balance to get new
> offsets so can you can tell me which dump tree you'd like so you
> hopefully get a non 0 result this time
>
> I'll reboot the system after that, btrfs check the FS, wait for your dumptree
> command, nad will run memtest overnight
> BTRFS critical (device dm-4): adding refs to an existing tree ref, bytenr 470809264128 num_bytes 16384 root_objectid 398 slot 1
> BTRFS error (device dm-4): failed to run delayed ref for logical 470809264128 num_bytes 16384 type 176 action 1 ref_mod 1: -117
Like last time, btrfs check came clean:
sauron:~# btrfs check --force /dev/mapper/pool1
Opening filesystem to check...
WARNING: filesystem mounted, continuing because of --force
Checking filesystem on /dev/mapper/pool1
UUID: 4542883b-d8bc-4d7f-8a2e-944dc355dc44
[1/7] checking root items
[2/7] checking extents
[3/7] checking free space tree
[4/7] checking fs roots
[5/7] checking only csums items (without verifying data)
[6/7] checking root refs
[7/7] checking quota groups skipped (not enabled on this FS)
found 225666609152 bytes used, no error found
total csum bytes: 216262104
total tree bytes: 4427300864
total fs tree bytes: 3598286848
total extent tree bytes: 490405888
btree space waste bytes: 1041370018
file data blocks allocated: 14364173885440
referenced 275881979904
sauron:~#
As a side note, is there a way to kill enough of systemd so that it will
stop using /var/log, which is preventing me from unmounting it and
running fsck without --force?
init 1 didn't do the trick
As for memcheck, I just run it 5mn for now, it seemed ok. I'm attaching
a screenshot so you can confirm it looks like the correct test and
correct version.
I will run it overnight tonight.
In the meantime, please let me know what new dump-tree I should run
Thanks,
Marc
--
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Home page: http://marc.merlins.org/
[-- Attachment #2: memcheck.jpg --]
[-- Type: image/jpeg, Size: 405055 bytes --]
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: BTRFS error (device dm-4): failed to run delayed ref for logical 350223581184 num_bytes 16384 type 176 action 1 ref_mod 1: -117 (kernel 6.11.2)
2025-02-06 17:07 BTRFS error (device dm-4): failed to run delayed ref for logical 350223581184 num_bytes 16384 type 176 action 1 ref_mod 1: -117 (kernel 6.11.2) Marc MERLIN
2025-02-06 17:34 ` Marc MERLIN
2025-02-25 7:32 ` Marc MERLIN
@ 2025-02-27 17:02 ` Boris Burkov
2025-02-27 17:21 ` Boris Burkov
2 siblings, 1 reply; 17+ messages in thread
From: Boris Burkov @ 2025-02-27 17:02 UTC (permalink / raw)
To: Marc MERLIN; +Cc: linux-btrfs
On Thu, Feb 06, 2025 at 09:07:31AM -0800, Marc MERLIN wrote:
> Balance ended with btrfs_run_delayed_refs:2199: errno=-117 Filesystem corrupted
>
> btrfs check says it's not
> sauron:~# btrfs check /dev/mapper/pool1
> Opening filesystem to check...
> Checking filesystem on /dev/mapper/pool1
> UUID: 4542883b-d8bc-4d7f-8a2e-944dc355dc44
> [1/7] checking root items
> [2/7] checking extents
> [3/7] checking free space tree
> [4/7] checking fs roots
> [5/7] checking only csums items (without verifying data)
> [6/7] checking root refs
> [7/7] checking quota groups skipped (not enabled on this FS)
> found 228820946944 bytes used, no error found
> total csum bytes: 219270232
> total tree bytes: 4539334656
> total fs tree bytes: 3719593984
> total extent tree bytes: 481935360
> btree space waste bytes: 1075469196
> file data blocks allocated: 15390875832320
> referenced 290833076224
>
> Any ideas? this obviously caused downtime, but after the btrfs check saying I'm supposedly
> ok, I'm back up for now, hoping it won't happen again and hope is not a strategy :)
I haven't carefully read the entire thread because it looks like it was
focused on memtest stuff, so apologies if I missed something.
This looks exactly like a bug Josef and I debugged extensively last
year.
https://lore.kernel.org/linux-btrfs/68766e66ed15ca2e7550585ed09434249db912a2.1727212293.git.josef@toxicpanda.com/
and
https://lore.kernel.org/linux-btrfs/fc61fb63e534111f5837c204ec341c876637af69.1731513908.git.josef@toxicpanda.com/
I'll dig through the rest of the emails now, confirm whether you have
the fixes, etc. but just wanted to get that on your radar.
Boris
>
> Is it safe to run balance again? any idea why it failed?
>
> [1821331.015652] BTRFS info (device dm-4): balance: start -dusage=20
> [1821331.015805] BTRFS info (device dm-4): relocating block group 761212698624 flags data
> [1821331.090338] BTRFS info (device dm-4): found 31 extents, stage: move data extents
> [1821331.237707] BTRFS info (device dm-4): leaf 471333519360 gen 4808182 total ptrs 168 free space 3533 owner 2
> [1821331.237716] item 0 key (350222417920 169 0) itemoff 16250 itemsize 33
> [1821331.237718] extent refs 1 gen 2907391 flags 2
> [1821331.237719] ref#0: tree block backref root 398
> (...(
> [1821331.238559] item 167 key (350225678336 169 0) itemoff 7733 itemsize 168
> [1821331.238560] extent refs 16 gen 4619087 flags 2
> [1821331.238561] ref#0: tree block backref root 398
> [1821331.238562] ref#1: shared block backref parent 737084112896
> [1821331.238563] ref#2: shared block backref parent 736609173504
> [1821331.238564] ref#3: shared block backref parent 471099588608
> [1821331.238565] ref#4: shared block backref parent 471017488384
> [1821331.238567] ref#5: shared block backref parent 470665625600
> [1821331.238568] ref#6: shared block backref parent 350806835200
> [1821331.238569] ref#7: shared block backref parent 350292066304
> [1821331.238570] ref#8: shared block backref parent 349856350208
> [1821331.238571] ref#9: shared block backref parent 153429573632
> [1821331.238572] ref#10: shared block backref parent 153014337536
> [1821331.238573] ref#11: shared block backref parent 152976048128
> [1821331.238575] ref#12: shared block backref parent 152753946624
> [1821331.238576] ref#13: shared block backref parent 152639225856
> [1821331.238577] ref#14: shared block backref parent 50782617600
> [1821331.238578] ref#15: shared block backref parent 394002432
> [1821331.238579] BTRFS critical (device dm-4): adding refs to an existing tree ref, bytenr 350223581184 num_bytes 16384 root_objectid 398 slot 51
> [1821331.238582] BTRFS error (device dm-4): failed to run delayed ref for logical 350223581184 num_bytes 16384 type 176 action 1 ref_mod 1: -117
> [1821331.238584] ------------[ cut here ]------------
> [1821331.238585] BTRFS: Transaction aborted (error -117)
> [1821331.238593] WARNING: CPU: 1 PID: 2457672 at fs/btrfs/extent-tree.c:2199 btrfs_run_delayed_refs+0x107/0x140
> [1821331.238599] Modules linked in: mmc_block exfat uinput rpcsec_gss_krb5 nfsv4 dns_resolver nfs netfs sg uas usb_storage nf_conntrack_netlink xfrm_user xfrm_algo xt_addrtype br_netfilter bridge stp llc xt_tcpudp xt_conntrack rfcomm snd_seq_dummy snd_hrtimer ccm overlay ipt_REJECT nf_reject_ipv4 xt_MASQUERADE xt_LOG nf_log_syslog nft_compat nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 cmac algif_hash algif_skcipher af_alg nf_tables bnep binfmt_misc uvcvideo videobuf2_vmalloc uvc videobuf2_memops btusb videobuf2_v4l2 btrtl btbcm videodev btmtk btintel videobuf2_common mc bluetooth nls_utf8 nls_cp437 vfat fat squashfs loop snd_hda_codec_hdmi iwlmvm snd_hda_codec_realtek snd_hda_codec_generic snd_soc_dmic snd_hda_scodec_component mac80211 intel_uncore_frequency snd_sof_pci_intel_tgl intel_uncore_frequency_common snd_sof_pci_intel_cnl intel_tcc_cooling snd_sof_intel_hda_generic snd_sof_intel_hda_common x86_pkg_temp_thermal snd_sof_intel_hda intel_powerclamp libarc4 snd_sof_pci snd_sof_xtensa_dsp kvm_intel
> [1821331.238644] snd_soc_hdac_hda iwlwifi kvm thinkpad_acpi snd_soc_acpi_intel_match mei_hdcp mei_pxp snd_soc_acpi cfg80211 processor_thermal_device_pci_legacy nvram tpm_crb processor_thermal_device rapl platform_profile processor_thermal_wt_hint processor_thermal_rfim snd_soc_avs processor_thermal_rapl ucsi_acpi intel_rapl_common nvidiafb processor_thermal_wt_req intel_cstate think_lmi vgastate iTCO_wdt typec_ucsi snd_soc_hda_codec pcspkr processor_thermal_power_floor firmware_attributes_class wmi_bmof snd_hda_intel ee1004 iTCO_vendor_support fb_ddc typec processor_thermal_mbox mei_me roles intel_soc_dts_iosf int3403_thermal rfkill int340x_thermal_zone ac intel_pmc_core intel_vsec tpm_tis tpm_tis_core int3400_thermal pmt_telemetry acpi_pad intel_hid acpi_thermal_rel sparse_keymap pmt_class acpi_tad input_leds evdev joydev serio_raw vboxdrv(OE) soundwire_intel soundwire_cadence snd_sof_intel_hda_mlink soundwire_generic_allocation snd_sof_probes snd_sof snd_sof_utils snd_intel_dspcfg snd_intel_sdw_acpi snd_soc_skl_hda_dsp
> [1821331.238682] snd_soc_intel_hda_dsp_common snd_hda_codec snd_hwdep snd_soc_hdac_hdmi snd_hda_ext_core snd_soc_core snd_compress snd_pcm_dmaengine snd_hda_core snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_midi snd_seq_midi_event snd_seq snd_timer snd_rawmidi snd_seq_device snd_ctl_led snd soundcore ac97_bus configs coretemp msr fuse efi_pstore nfsd auth_rpcgss nfs_acl lockd grace sunrpc nfnetlink ip_tables x_tables autofs4 essiv authenc dm_crypt trusted asn1_encoder tee tpm rng_core libaescfb ecdh_generic dm_mod ecc raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx sata_sil24 r8169 realtek mdio_devres libphy mii hid_generic usbhid hid i915 crct10dif_pclmul drm_buddy i2c_algo_bit rtsx_pci_sdmmc crc32_pclmul xhci_pci ttm mmc_core crc32c_intel xhci_hcd polyval_clmulni drm_display_helper polyval_generic usbcore cec i2c_i801 rc_core video ptp spi_intel_pci i2c_mux ghash_clmulni_intel sha512_ssse3 sha256_ssse3 sha1_ssse3 psmouse thunderbolt rtsx_pci spi_intel i2c_smbus pps_core usb_common thermal hwmon battery
> [1821331.238731] wmi aesni_intel crypto_simd cryptd [last unloaded: igc]
> [1821331.238737] CPU: 1 UID: 0 PID: 2457672 Comm: btrfs Tainted: G U OE 6.11.2-amd64-preempt-sysrq-20241007 #1 1a512c2db5f087f236d90ecfb30551fddcc51243
> [1821331.238740] Tainted: [U]=USER, [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
> [1821331.238742] Hardware name: LENOVO 20YU002JUS/20YU002JUS, BIOS N37ET49W (1.30 ) 11/15/2023
> [1821331.238743] RIP: 0010:btrfs_run_delayed_refs+0x107/0x140
> [1821331.238745] Code: 01 00 00 00 eb b6 e8 18 8e b7 00 31 db 89 d8 5b 5d 41 5c 41 5d 41 5e c3 cc cc cc cc 89 de 48 c7 c7 40 bb b7 86 e8 f9 0c 9f ff <0f> 0b eb d0 66 66 2e 0f 1f 84 00 00 00 00 00 66 66 2e 0f 1f 84 00
> [1821331.238747] RSP: 0018:ffffaf1a6f167858 EFLAGS: 00010282
> [1821331.238749] RAX: 0000000000000000 RBX: 00000000ffffff8b RCX: 0000000000000027
> [1821331.238750] RDX: ffff8fcf2f2a1848 RSI: 0000000000000001 RDI: ffff8fcf2f2a1840
> [1821331.238752] RBP: ffff8fb06efb8150 R08: 0000000000000000 R09: 0000000000000003
> [1821331.238753] R10: ffffaf1a6f1676f8 R11: ffff8fcfaf7d5028 R12: 0000000000000000
> [1821331.238754] R13: ffff8fb146731358 R14: ffff8fb146731200 R15: 0000000000000000
> [1821331.238755] FS: 00007fd91883d380(0000) GS:ffff8fcf2f280000(0000) knlGS:0000000000000000
> [1821331.238756] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [1821331.238757] CR2: 00007e4901001000 CR3: 00000001c3cb6004 CR4: 0000000000770ef0
> [1821331.238759] PKRU: 55555554
> [1821331.238760] Call Trace:
> [1821331.238762] <TASK>
> [1821331.238765] ? __warn+0x7c/0x140
> [1821331.238769] ? btrfs_run_delayed_refs+0x107/0x140
> [1821331.238771] ? report_bug+0x160/0x1c0
> [1821331.238774] ? handle_bug+0x41/0x80
> [1821331.238777] ? exc_invalid_op+0x15/0x100
> [1821331.238780] ? asm_exc_invalid_op+0x16/0x40
> [1821331.238783] ? btrfs_run_delayed_refs+0x107/0x140
> [1821331.238785] ? btrfs_run_delayed_refs+0x107/0x140
> [1821331.238786] btrfs_commit_transaction+0x69/0xe80
> [1821331.238790] ? btrfs_update_reloc_root+0x12d/0x240
> [1821331.238793] prepare_to_merge+0x4f0/0x600
> [1821331.238796] relocate_block_group+0x113/0x500
> [1821331.238798] btrfs_relocate_block_group+0x27a/0x440
> [1821331.238800] btrfs_relocate_chunk+0x3b/0x180
> [1821331.238803] btrfs_balance+0x8c1/0x1340
> [1821331.238805] ? btrfs_ioctl+0x18db/0x26c0
> [1821331.238811] btrfs_ioctl+0x2285/0x26c0
> [1821331.238813] ? __mod_memcg_lruvec_state+0x91/0x140
> [1821331.238817] ? vsnprintf+0x323/0x580
> [1821331.238819] ? __slab_free+0x53/0x2c0
> [1821331.238822] ? sysfs_emit+0x68/0xc0
> [1821331.238826] __x64_sys_ioctl+0x90/0x100
> [1821331.238830] do_syscall_64+0x69/0x140
> [1821331.238832] ? __memcg_slab_free_hook+0xf3/0x140
> [1821331.238835] ? __x64_sys_close+0x38/0x80
> [1821331.238838] ? kmem_cache_free+0x336/0x400
> [1821331.238840] ? do_syscall_64+0x75/0x140
> [1821331.238842] ? ksys_read+0x63/0x100
> [1821331.238845] ? __mod_memcg_lruvec_state+0x91/0x140
> [1821331.238848] ? mod_objcg_state+0x19d/0x2c0
> [1821331.238850] ? __memcg_slab_free_hook+0xf3/0x140
> [1821331.238852] ? seq_release+0x24/0x40
> [1821331.238854] ? __memcg_slab_free_hook+0xf3/0x140
> [1821331.238856] ? __x64_sys_close+0x38/0x80
> [1821331.238858] ? kmem_cache_free+0x336/0x400
> [1821331.238860] ? clear_bhb_loop+0x45/0xc0
> [1821331.238862] ? clear_bhb_loop+0x45/0xc0
> [1821331.238864] ? clear_bhb_loop+0x45/0xc0
> [1821331.238866] ? clear_bhb_loop+0x45/0xc0
> [1821331.238867] ? clear_bhb_loop+0x45/0xc0
> [1821331.238869] entry_SYSCALL_64_after_hwframe+0x76/0x7e
> [1821331.238872] RIP: 0033:0x7fd9189564bb
> [1821331.238874] Code: 00 48 89 44 24 18 31 c0 48 8d 44 24 60 c7 04 24 10 00 00 00 48 89 44 24 08 48 8d 44 24 20 48 89 44 24 10 b8 10 00 00 00 0f 05 <89> c2 3d 00 f0 ff ff 77 1c 48 8b 44 24 18 64 48 2b 04 25 28 00 00
> [1821331.238876] RSP: 002b:00007ffd5896d7a0 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
> [1821331.238878] RAX: ffffffffffffffda RBX: 0000000000000003 RCX: 00007fd9189564bb
> [1821331.238879] RDX: 00007ffd5896d8a8 RSI: 00000000c4009420 RDI: 0000000000000003
> [1821331.238880] RBP: 0000000000000000 R08: 0000000000000073 R09: 0000000000000013
> [1821331.238881] R10: 0000000000000000 R11: 0000000000000246 R12: 00007ffd5896d8a8
> [1821331.238882] R13: 0000000000000000 R14: 00007ffd5896ee24 R15: 0000000000000001
> [1821331.238884] </TASK>
> [1821331.238885] ---[ end trace 0000000000000000 ]---
> [1821331.238886] BTRFS: error (device dm-4 state A) in btrfs_run_delayed_refs:2199: errno=-117 Filesystem corrupted
>
> --
> "A mouse is a device used to point at the xterm you want to type in" - A.S.R.
>
> Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: BTRFS error (device dm-4): failed to run delayed ref for logical 350223581184 num_bytes 16384 type 176 action 1 ref_mod 1: -117 (kernel 6.11.2)
2025-02-27 17:02 ` Boris Burkov
@ 2025-02-27 17:21 ` Boris Burkov
2025-02-27 17:33 ` Marc MERLIN
0 siblings, 1 reply; 17+ messages in thread
From: Boris Burkov @ 2025-02-27 17:21 UTC (permalink / raw)
To: Marc MERLIN; +Cc: linux-btrfs
On Thu, Feb 27, 2025 at 09:02:20AM -0800, Boris Burkov wrote:
> On Thu, Feb 06, 2025 at 09:07:31AM -0800, Marc MERLIN wrote:
> > Balance ended with btrfs_run_delayed_refs:2199: errno=-117 Filesystem corrupted
> >
> > btrfs check says it's not
> > sauron:~# btrfs check /dev/mapper/pool1
> > Opening filesystem to check...
> > Checking filesystem on /dev/mapper/pool1
> > UUID: 4542883b-d8bc-4d7f-8a2e-944dc355dc44
> > [1/7] checking root items
> > [2/7] checking extents
> > [3/7] checking free space tree
> > [4/7] checking fs roots
> > [5/7] checking only csums items (without verifying data)
> > [6/7] checking root refs
> > [7/7] checking quota groups skipped (not enabled on this FS)
> > found 228820946944 bytes used, no error found
> > total csum bytes: 219270232
> > total tree bytes: 4539334656
> > total fs tree bytes: 3719593984
> > total extent tree bytes: 481935360
> > btree space waste bytes: 1075469196
> > file data blocks allocated: 15390875832320
> > referenced 290833076224
> >
> > Any ideas? this obviously caused downtime, but after the btrfs check saying I'm supposedly
> > ok, I'm back up for now, hoping it won't happen again and hope is not a strategy :)
>
> I haven't carefully read the entire thread because it looks like it was
> focused on memtest stuff, so apologies if I missed something.
>
> This looks exactly like a bug Josef and I debugged extensively last
> year.
>
> https://lore.kernel.org/linux-btrfs/68766e66ed15ca2e7550585ed09434249db912a2.1727212293.git.josef@toxicpanda.com/
> and
> https://lore.kernel.org/linux-btrfs/fc61fb63e534111f5837c204ec341c876637af69.1731513908.git.josef@toxicpanda.com/
>
> I'll dig through the rest of the emails now, confirm whether you have
> the fixes, etc. but just wanted to get that on your radar.
Your kernel is 6.11.2 and those fixes went in to 6.12-rc2:
https://lore.kernel.org/linux-btrfs/cover.1728050979.git.dsterba@suse.com/
and 6.12-rc8:
https://lore.kernel.org/linux-btrfs/cover.1731619157.git.dsterba@suse.com/
so unless you have backported those fixes, that is almost certainly the issue.
>
> Boris
>
> >
> > Is it safe to run balance again? any idea why it failed?
> >
> > [1821331.015652] BTRFS info (device dm-4): balance: start -dusage=20
> > [1821331.015805] BTRFS info (device dm-4): relocating block group 761212698624 flags data
> > [1821331.090338] BTRFS info (device dm-4): found 31 extents, stage: move data extents
> > [1821331.237707] BTRFS info (device dm-4): leaf 471333519360 gen 4808182 total ptrs 168 free space 3533 owner 2
> > [1821331.237716] item 0 key (350222417920 169 0) itemoff 16250 itemsize 33
> > [1821331.237718] extent refs 1 gen 2907391 flags 2
> > [1821331.237719] ref#0: tree block backref root 398
> > (...(
> > [1821331.238559] item 167 key (350225678336 169 0) itemoff 7733 itemsize 168
> > [1821331.238560] extent refs 16 gen 4619087 flags 2
> > [1821331.238561] ref#0: tree block backref root 398
> > [1821331.238562] ref#1: shared block backref parent 737084112896
> > [1821331.238563] ref#2: shared block backref parent 736609173504
> > [1821331.238564] ref#3: shared block backref parent 471099588608
> > [1821331.238565] ref#4: shared block backref parent 471017488384
> > [1821331.238567] ref#5: shared block backref parent 470665625600
> > [1821331.238568] ref#6: shared block backref parent 350806835200
> > [1821331.238569] ref#7: shared block backref parent 350292066304
> > [1821331.238570] ref#8: shared block backref parent 349856350208
> > [1821331.238571] ref#9: shared block backref parent 153429573632
> > [1821331.238572] ref#10: shared block backref parent 153014337536
> > [1821331.238573] ref#11: shared block backref parent 152976048128
> > [1821331.238575] ref#12: shared block backref parent 152753946624
> > [1821331.238576] ref#13: shared block backref parent 152639225856
> > [1821331.238577] ref#14: shared block backref parent 50782617600
> > [1821331.238578] ref#15: shared block backref parent 394002432
> > [1821331.238579] BTRFS critical (device dm-4): adding refs to an existing tree ref, bytenr 350223581184 num_bytes 16384 root_objectid 398 slot 51
> > [1821331.238582] BTRFS error (device dm-4): failed to run delayed ref for logical 350223581184 num_bytes 16384 type 176 action 1 ref_mod 1: -117
> > [1821331.238584] ------------[ cut here ]------------
> > [1821331.238585] BTRFS: Transaction aborted (error -117)
> > [1821331.238593] WARNING: CPU: 1 PID: 2457672 at fs/btrfs/extent-tree.c:2199 btrfs_run_delayed_refs+0x107/0x140
> > [1821331.238599] Modules linked in: mmc_block exfat uinput rpcsec_gss_krb5 nfsv4 dns_resolver nfs netfs sg uas usb_storage nf_conntrack_netlink xfrm_user xfrm_algo xt_addrtype br_netfilter bridge stp llc xt_tcpudp xt_conntrack rfcomm snd_seq_dummy snd_hrtimer ccm overlay ipt_REJECT nf_reject_ipv4 xt_MASQUERADE xt_LOG nf_log_syslog nft_compat nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 cmac algif_hash algif_skcipher af_alg nf_tables bnep binfmt_misc uvcvideo videobuf2_vmalloc uvc videobuf2_memops btusb videobuf2_v4l2 btrtl btbcm videodev btmtk btintel videobuf2_common mc bluetooth nls_utf8 nls_cp437 vfat fat squashfs loop snd_hda_codec_hdmi iwlmvm snd_hda_codec_realtek snd_hda_codec_generic snd_soc_dmic snd_hda_scodec_component mac80211 intel_uncore_frequency snd_sof_pci_intel_tgl intel_uncore_frequency_common snd_sof_pci_intel_cnl intel_tcc_cooling snd_sof_intel_hda_generic snd_sof_intel_hda_common x86_pkg_temp_thermal snd_sof_intel_hda intel_powerclamp libarc4 snd_sof_pci snd_sof_xtensa_dsp kvm_intel
> > [1821331.238644] snd_soc_hdac_hda iwlwifi kvm thinkpad_acpi snd_soc_acpi_intel_match mei_hdcp mei_pxp snd_soc_acpi cfg80211 processor_thermal_device_pci_legacy nvram tpm_crb processor_thermal_device rapl platform_profile processor_thermal_wt_hint processor_thermal_rfim snd_soc_avs processor_thermal_rapl ucsi_acpi intel_rapl_common nvidiafb processor_thermal_wt_req intel_cstate think_lmi vgastate iTCO_wdt typec_ucsi snd_soc_hda_codec pcspkr processor_thermal_power_floor firmware_attributes_class wmi_bmof snd_hda_intel ee1004 iTCO_vendor_support fb_ddc typec processor_thermal_mbox mei_me roles intel_soc_dts_iosf int3403_thermal rfkill int340x_thermal_zone ac intel_pmc_core intel_vsec tpm_tis tpm_tis_core int3400_thermal pmt_telemetry acpi_pad intel_hid acpi_thermal_rel sparse_keymap pmt_class acpi_tad input_leds evdev joydev serio_raw vboxdrv(OE) soundwire_intel soundwire_cadence snd_sof_intel_hda_mlink soundwire_generic_allocation snd_sof_probes snd_sof snd_sof_utils snd_intel_dspcfg snd_intel_sdw_acpi snd_soc_skl_hda_dsp
> > [1821331.238682] snd_soc_intel_hda_dsp_common snd_hda_codec snd_hwdep snd_soc_hdac_hdmi snd_hda_ext_core snd_soc_core snd_compress snd_pcm_dmaengine snd_hda_core snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_midi snd_seq_midi_event snd_seq snd_timer snd_rawmidi snd_seq_device snd_ctl_led snd soundcore ac97_bus configs coretemp msr fuse efi_pstore nfsd auth_rpcgss nfs_acl lockd grace sunrpc nfnetlink ip_tables x_tables autofs4 essiv authenc dm_crypt trusted asn1_encoder tee tpm rng_core libaescfb ecdh_generic dm_mod ecc raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx sata_sil24 r8169 realtek mdio_devres libphy mii hid_generic usbhid hid i915 crct10dif_pclmul drm_buddy i2c_algo_bit rtsx_pci_sdmmc crc32_pclmul xhci_pci ttm mmc_core crc32c_intel xhci_hcd polyval_clmulni drm_display_helper polyval_generic usbcore cec i2c_i801 rc_core video ptp spi_intel_pci i2c_mux ghash_clmulni_intel sha512_ssse3 sha256_ssse3 sha1_ssse3 psmouse thunderbolt rtsx_pci spi_intel i2c_smbus pps_core usb_common thermal hwmon battery
> > [1821331.238731] wmi aesni_intel crypto_simd cryptd [last unloaded: igc]
> > [1821331.238737] CPU: 1 UID: 0 PID: 2457672 Comm: btrfs Tainted: G U OE 6.11.2-amd64-preempt-sysrq-20241007 #1 1a512c2db5f087f236d90ecfb30551fddcc51243
> > [1821331.238740] Tainted: [U]=USER, [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
> > [1821331.238742] Hardware name: LENOVO 20YU002JUS/20YU002JUS, BIOS N37ET49W (1.30 ) 11/15/2023
> > [1821331.238743] RIP: 0010:btrfs_run_delayed_refs+0x107/0x140
> > [1821331.238745] Code: 01 00 00 00 eb b6 e8 18 8e b7 00 31 db 89 d8 5b 5d 41 5c 41 5d 41 5e c3 cc cc cc cc 89 de 48 c7 c7 40 bb b7 86 e8 f9 0c 9f ff <0f> 0b eb d0 66 66 2e 0f 1f 84 00 00 00 00 00 66 66 2e 0f 1f 84 00
> > [1821331.238747] RSP: 0018:ffffaf1a6f167858 EFLAGS: 00010282
> > [1821331.238749] RAX: 0000000000000000 RBX: 00000000ffffff8b RCX: 0000000000000027
> > [1821331.238750] RDX: ffff8fcf2f2a1848 RSI: 0000000000000001 RDI: ffff8fcf2f2a1840
> > [1821331.238752] RBP: ffff8fb06efb8150 R08: 0000000000000000 R09: 0000000000000003
> > [1821331.238753] R10: ffffaf1a6f1676f8 R11: ffff8fcfaf7d5028 R12: 0000000000000000
> > [1821331.238754] R13: ffff8fb146731358 R14: ffff8fb146731200 R15: 0000000000000000
> > [1821331.238755] FS: 00007fd91883d380(0000) GS:ffff8fcf2f280000(0000) knlGS:0000000000000000
> > [1821331.238756] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > [1821331.238757] CR2: 00007e4901001000 CR3: 00000001c3cb6004 CR4: 0000000000770ef0
> > [1821331.238759] PKRU: 55555554
> > [1821331.238760] Call Trace:
> > [1821331.238762] <TASK>
> > [1821331.238765] ? __warn+0x7c/0x140
> > [1821331.238769] ? btrfs_run_delayed_refs+0x107/0x140
> > [1821331.238771] ? report_bug+0x160/0x1c0
> > [1821331.238774] ? handle_bug+0x41/0x80
> > [1821331.238777] ? exc_invalid_op+0x15/0x100
> > [1821331.238780] ? asm_exc_invalid_op+0x16/0x40
> > [1821331.238783] ? btrfs_run_delayed_refs+0x107/0x140
> > [1821331.238785] ? btrfs_run_delayed_refs+0x107/0x140
> > [1821331.238786] btrfs_commit_transaction+0x69/0xe80
> > [1821331.238790] ? btrfs_update_reloc_root+0x12d/0x240
> > [1821331.238793] prepare_to_merge+0x4f0/0x600
> > [1821331.238796] relocate_block_group+0x113/0x500
> > [1821331.238798] btrfs_relocate_block_group+0x27a/0x440
> > [1821331.238800] btrfs_relocate_chunk+0x3b/0x180
> > [1821331.238803] btrfs_balance+0x8c1/0x1340
> > [1821331.238805] ? btrfs_ioctl+0x18db/0x26c0
> > [1821331.238811] btrfs_ioctl+0x2285/0x26c0
> > [1821331.238813] ? __mod_memcg_lruvec_state+0x91/0x140
> > [1821331.238817] ? vsnprintf+0x323/0x580
> > [1821331.238819] ? __slab_free+0x53/0x2c0
> > [1821331.238822] ? sysfs_emit+0x68/0xc0
> > [1821331.238826] __x64_sys_ioctl+0x90/0x100
> > [1821331.238830] do_syscall_64+0x69/0x140
> > [1821331.238832] ? __memcg_slab_free_hook+0xf3/0x140
> > [1821331.238835] ? __x64_sys_close+0x38/0x80
> > [1821331.238838] ? kmem_cache_free+0x336/0x400
> > [1821331.238840] ? do_syscall_64+0x75/0x140
> > [1821331.238842] ? ksys_read+0x63/0x100
> > [1821331.238845] ? __mod_memcg_lruvec_state+0x91/0x140
> > [1821331.238848] ? mod_objcg_state+0x19d/0x2c0
> > [1821331.238850] ? __memcg_slab_free_hook+0xf3/0x140
> > [1821331.238852] ? seq_release+0x24/0x40
> > [1821331.238854] ? __memcg_slab_free_hook+0xf3/0x140
> > [1821331.238856] ? __x64_sys_close+0x38/0x80
> > [1821331.238858] ? kmem_cache_free+0x336/0x400
> > [1821331.238860] ? clear_bhb_loop+0x45/0xc0
> > [1821331.238862] ? clear_bhb_loop+0x45/0xc0
> > [1821331.238864] ? clear_bhb_loop+0x45/0xc0
> > [1821331.238866] ? clear_bhb_loop+0x45/0xc0
> > [1821331.238867] ? clear_bhb_loop+0x45/0xc0
> > [1821331.238869] entry_SYSCALL_64_after_hwframe+0x76/0x7e
> > [1821331.238872] RIP: 0033:0x7fd9189564bb
> > [1821331.238874] Code: 00 48 89 44 24 18 31 c0 48 8d 44 24 60 c7 04 24 10 00 00 00 48 89 44 24 08 48 8d 44 24 20 48 89 44 24 10 b8 10 00 00 00 0f 05 <89> c2 3d 00 f0 ff ff 77 1c 48 8b 44 24 18 64 48 2b 04 25 28 00 00
> > [1821331.238876] RSP: 002b:00007ffd5896d7a0 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
> > [1821331.238878] RAX: ffffffffffffffda RBX: 0000000000000003 RCX: 00007fd9189564bb
> > [1821331.238879] RDX: 00007ffd5896d8a8 RSI: 00000000c4009420 RDI: 0000000000000003
> > [1821331.238880] RBP: 0000000000000000 R08: 0000000000000073 R09: 0000000000000013
> > [1821331.238881] R10: 0000000000000000 R11: 0000000000000246 R12: 00007ffd5896d8a8
> > [1821331.238882] R13: 0000000000000000 R14: 00007ffd5896ee24 R15: 0000000000000001
> > [1821331.238884] </TASK>
> > [1821331.238885] ---[ end trace 0000000000000000 ]---
> > [1821331.238886] BTRFS: error (device dm-4 state A) in btrfs_run_delayed_refs:2199: errno=-117 Filesystem corrupted
> >
> > --
> > "A mouse is a device used to point at the xterm you want to type in" - A.S.R.
> >
> > Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: BTRFS error (device dm-4): failed to run delayed ref for logical 350223581184 num_bytes 16384 type 176 action 1 ref_mod 1: -117 (kernel 6.11.2)
2025-02-27 17:21 ` Boris Burkov
@ 2025-02-27 17:33 ` Marc MERLIN
2025-02-27 17:41 ` Filipe Manana
0 siblings, 1 reply; 17+ messages in thread
From: Marc MERLIN @ 2025-02-27 17:33 UTC (permalink / raw)
To: Boris Burkov, Josef Bacik, QuWenruo, Qu Wenruo
Cc: linux-btrfs, Josef Bacik, Chris Murphy, Zygo Blaxell,
Roman Mamedov
On Thu, Feb 27, 2025 at 09:21:13AM -0800, Boris Burkov wrote:
> > https://lore.kernel.org/linux-btrfs/68766e66ed15ca2e7550585ed09434249db912a2.1727212293.git.josef@toxicpanda.com/
> > and
> > https://lore.kernel.org/linux-btrfs/fc61fb63e534111f5837c204ec341c876637af69.1731513908.git.josef@toxicpanda.com/
> >
> > I'll dig through the rest of the emails now, confirm whether you have
> > the fixes, etc. but just wanted to get that on your radar.
>
> Your kernel is 6.11.2 and those fixes went in to 6.12-rc2:
> https://lore.kernel.org/linux-btrfs/cover.1728050979.git.dsterba@suse.com/
> and 6.12-rc8:
> https://lore.kernel.org/linux-btrfs/cover.1731619157.git.dsterba@suse.com/
>
> so unless you have backported those fixes, that is almost certainly the issue.
Thanks much Boris, I was not aware of this bug or patch, so indeed I
will need to merge in the patch.
Not sure if the 6.11 tree is closed or if the patch can also be added
there. Cc'ing Joseph.
I will merge the patch and report back since I seem to be able to
reproduce easily.
Marc
--
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Home page: http://marc.merlins.org/
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: BTRFS error (device dm-4): failed to run delayed ref for logical 350223581184 num_bytes 16384 type 176 action 1 ref_mod 1: -117 (kernel 6.11.2)
2025-02-27 17:33 ` Marc MERLIN
@ 2025-02-27 17:41 ` Filipe Manana
2025-02-27 19:57 ` Marc MERLIN
0 siblings, 1 reply; 17+ messages in thread
From: Filipe Manana @ 2025-02-27 17:41 UTC (permalink / raw)
To: Marc MERLIN
Cc: Boris Burkov, Josef Bacik, QuWenruo, Qu Wenruo, linux-btrfs,
Chris Murphy, Zygo Blaxell, Roman Mamedov
On Thu, Feb 27, 2025 at 5:33 PM Marc MERLIN <marc@merlins.org> wrote:
>
> On Thu, Feb 27, 2025 at 09:21:13AM -0800, Boris Burkov wrote:
> > > https://lore.kernel.org/linux-btrfs/68766e66ed15ca2e7550585ed09434249db912a2.1727212293.git.josef@toxicpanda.com/
> > > and
> > > https://lore.kernel.org/linux-btrfs/fc61fb63e534111f5837c204ec341c876637af69.1731513908.git.josef@toxicpanda.com/
> > >
> > > I'll dig through the rest of the emails now, confirm whether you have
> > > the fixes, etc. but just wanted to get that on your radar.
> >
> > Your kernel is 6.11.2 and those fixes went in to 6.12-rc2:
> > https://lore.kernel.org/linux-btrfs/cover.1728050979.git.dsterba@suse.com/
> > and 6.12-rc8:
> > https://lore.kernel.org/linux-btrfs/cover.1731619157.git.dsterba@suse.com/
> >
> > so unless you have backported those fixes, that is almost certainly the issue.
>
> Thanks much Boris, I was not aware of this bug or patch, so indeed I
> will need to merge in the patch.
>
> Not sure if the 6.11 tree is closed or if the patch can also be added
> there. Cc'ing Joseph.
>
> I will merge the patch and report back since I seem to be able to
> reproduce easily.
The patch was backported to 6.11.3, you're on 6.11.2... can't you
update the kernel to 6.11.11 (the last 6.11 stable kernel)?
>
> Marc
> --
> "A mouse is a device used to point at the xterm you want to type in" - A.S.R.
>
> Home page: http://marc.merlins.org/
>
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: BTRFS error (device dm-4): failed to run delayed ref for logical 350223581184 num_bytes 16384 type 176 action 1 ref_mod 1: -117 (kernel 6.11.2)
2025-02-27 17:41 ` Filipe Manana
@ 2025-02-27 19:57 ` Marc MERLIN
0 siblings, 0 replies; 17+ messages in thread
From: Marc MERLIN @ 2025-02-27 19:57 UTC (permalink / raw)
To: Filipe Manana
Cc: Boris Burkov, Josef Bacik, QuWenruo, Qu Wenruo, linux-btrfs,
Chris Murphy, Zygo Blaxell, Roman Mamedov
On Thu, Feb 27, 2025 at 05:41:29PM +0000, Filipe Manana wrote:
> > I will merge the patch and report back since I seem to be able to
> > reproduce easily.
>
> The patch was backported to 6.11.3, you're on 6.11.2... can't you
> update the kernel to 6.11.11 (the last 6.11 stable kernel)?
I totally can, I didn't know what the bug was, or which kernels it was
fixed in, and don't have time to upgrade to every point release unless
there is a need, which there is in this case.
Will upgrade now, thanks much for the guidance, much appreciated.
Marc
--
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Home page: http://marc.merlins.org/
^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: BTRFS error (device dm-4): failed to run delayed ref for logical 350223581184 num_bytes 16384 type 176 action 1 ref_mod 1: -117 (kernel 6.11.2)
2025-02-27 14:11 ` Marc MERLIN
2025-02-27 15:05 ` Marc MERLIN
@ 2025-02-28 13:10 ` Marc MERLIN
1 sibling, 0 replies; 17+ messages in thread
From: Marc MERLIN @ 2025-02-28 13:10 UTC (permalink / raw)
To: Qu Wenruo
Cc: Qu Wenruo, linux-btrfs, Josef Bacik, Chris Murphy, Zygo Blaxell,
Roman Mamedov, Su Yue
Since I was told it was a known and fixed bug (my apologies for not
knowing that), I upgraded from 6.11.2 to 6.13.5 and the bug seems fixed
now.
Thanks for all those who helped.
Marc
--
Marc
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
^ permalink raw reply [flat|nested] 17+ messages in thread
end of thread, other threads:[~2025-02-28 13:10 UTC | newest]
Thread overview: 17+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2025-02-06 17:07 BTRFS error (device dm-4): failed to run delayed ref for logical 350223581184 num_bytes 16384 type 176 action 1 ref_mod 1: -117 (kernel 6.11.2) Marc MERLIN
2025-02-06 17:34 ` Marc MERLIN
2025-02-25 7:32 ` Marc MERLIN
2025-02-25 8:07 ` Qu Wenruo
2025-02-25 8:29 ` Marc MERLIN
2025-02-25 8:46 ` Qu Wenruo
2025-02-25 9:18 ` Marc MERLIN
2025-02-25 9:22 ` Qu Wenruo
2025-02-27 14:11 ` Marc MERLIN
2025-02-27 15:05 ` Marc MERLIN
2025-02-28 13:10 ` Marc MERLIN
2025-02-25 9:25 ` Roman Mamedov
2025-02-27 17:02 ` Boris Burkov
2025-02-27 17:21 ` Boris Burkov
2025-02-27 17:33 ` Marc MERLIN
2025-02-27 17:41 ` Filipe Manana
2025-02-27 19:57 ` Marc MERLIN
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox