out of space (>865 GB free), filesystem remounts read-only - how to recover?

[Tomasz Chmielewski <mangoo@wpkg.org>]

My 44 TB filesystem was getting full (~865 GB free out of 44 TB in 
total), so I thought removing some old data would be a good thing to do.

Unfortunately now I'm not able to recover - the filesystem remounts 
read-only (no free space in metadata) a few minutes after mounting.

- kernel is 6.10.6

- the filesystem was writable and did not have any problems

- I've decided to delete some subvolumes with "btrfs sub del ..."

- a couple of minutes later after deleting the subvolumes, the 
filesystem went read only

- when I unmount and mount again - it resumes deleting the subvolumes in 
the background, free space goes up from 865G to 870G and filesystem 
remounts read-only

- after unmounting/mounting again, it again has 865GB free

- even if I remove some files right after mounting, as soon as the 
filesystem goes read-only - I unmount/mount again - removed files 
magically appear again

- I've added a 100 GB device to the filesystem - but it doesn't help, 
the filesystem still goes read only shortly after mounting

- tried running data and metadata balance - but the filesystem still 
goes read only shortly after mounting

How can I recover from that?

1) dmesg after mounting with "skip_balance" option:

[26883.769459] BTRFS info (device sdc1 state EA): last unmount of 
filesystem a80ce575-8c39-4065-80ce-2ca015fa1d51
[26890.241370] BTRFS info (device sdc1): first mount of filesystem 
a80ce575-8c39-4065-80ce-2ca015fa1d51
[26890.241402] BTRFS info (device sdc1): using crc32c (crc32c-intel) 
checksum algorithm
[26890.241414] BTRFS info (device sdc1): using free-space-tree
[27025.188366] BTRFS info (device sdc1): balance: resume skipped


2) dmesg after it goes read-only:

[27493.879003] ------------[ cut here ]------------
[27493.879014] BTRFS: Transaction aborted (error -28)
[27493.887226] BTRFS: error (device sdc1 state A) in 
do_free_extent_accounting:2968: errno=-28 No space left
[27493.936410] WARNING: CPU: 7 PID: 2990 at fs/btrfs/extent-tree.c:2968 
do_free_extent_accounting.cold+0xec/0x1f0 [btrfs]
[27494.050834] BTRFS info (device sdc1 state EA): forced readonly
[27494.050841] BTRFS error (device sdc1 state EA): failed to run delayed 
ref for logical 7329684533248 num_bytes 53248 type 184 action 2 ref_mod 
1: -28
[27494.051372] Modules linked in: tls
[27494.052220] BTRFS: error (device sdc1 state EA) in 
btrfs_run_delayed_refs:2207: errno=-28 No space left
[27494.211400]  isofs binfmt_misc ip6t_REJECT nf_reject_ipv6 
nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 
ipt_REJECT nf_reject_ipv4 xt_tcpudp nft_compat nf_tables nfnetlink 
nls_iso8859_1 intel_rapl_msr intel_rapl_common intel_uncore_frequency 
intel_uncore_frequency_common sb_edac x86_pkg_temp_thermal 
intel_powerclamp coretemp kvm_intel ipmi_ssif kvm rapl intel_cstate 
cmdlinepart spi_nor input_leds mei_me joydev intel_pch_thermal mtd mei 
ioatdma acpi_ipmi mac_hid ipmi_si ipmi_devintf ipmi_msghandler acpi_pad 
sch_fq_codel dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua msr 
efi_pstore ip_tables x_tables autofs4 btrfs blake2b_generic raid10 
raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor 
raid6_pq libcrc32c raid0 raid1 hid_generic usbhid hid crct10dif_pclmul 
crc32_pclmul polyval_clmulni spi_intel_platform polyval_generic ixgbe 
ghash_clmulni_intel spi_intel ahci gpio_ich igb i2c_i801 xfrm_algo 
sha256_ssse3 mxm_wmi i2c_mux libahci lpc_ich sha1_ssse3 i2c_smbus 
i2c_algo_bit xhci_pci
[27494.325621]  dca xhci_pci_renesas mdio wmi aesni_intel crypto_simd 
cryptd
[27494.325631] CPU: 7 PID: 2990 Comm: btrfs-transacti Tainted: G        
W          6.10.6-061006-generic #202408190440
[27494.325635] Hardware name: Supermicro Super Server/X10SDV-TLN4F, BIOS 
2.1.v1 06/05/2020
[27494.325637] RIP: 0010:do_free_extent_accounting.cold+0xec/0x1f0 
[btrfs]
[27494.325721] Code: 83 e0 01 e8 ec 52 00 00 e9 d0 5f f1 ff 89 df e8 e0 
ee ff ff 84 c0 0f 84 e8 00 00 00 89 de 48 c7 c7 d0 4f a4 c0 e8 4a c4 b7 
fb <0f> 0b eb bc 45 31 c0 41 83 e0 01 89 d9 ba b1 0b 00 00 4c 89 e7 48
[27494.325724] RSP: 0018:ffff956e01fbbb50 EFLAGS: 00010246
[27494.325727] RAX: 0000000000000000 RBX: 00000000ffffffe4 RCX: 
0000000000000000
[27494.325729] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 
0000000000000000
[27494.325731] RBP: ffff956e01fbbb80 R08: 0000000000000000 R09: 
0000000000000000
[27494.325733] R10: 0000000000000000 R11: 0000000000000000 R12: 
ffff8a4135049f18
[27494.325735] R13: 000006aa92150000 R14: 0000000000010000 R15: 
ffff956e01fbbc08
[27494.325738] FS:  0000000000000000(0000) GS:ffff8a479f980000(0000) 
knlGS:0000000000000000
[27494.325740] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[27494.325742] CR2: 0000713914aeef04 CR3: 000000016223c002 CR4: 
00000000003706f0
[27494.325745] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 
0000000000000000
[27494.325747] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 
0000000000000400
[27494.325749] Call Trace:
[27494.325751]  <TASK>
[27494.325754]  ? show_trace_log_lvl+0x1be/0x310
[27494.325761]  ? show_trace_log_lvl+0x1be/0x310
[27494.325766]  ? __btrfs_free_extent.isra.0+0x62a/0x900 [btrfs]
[27494.325838]  ? show_regs.part.0+0x22/0x30
[27494.325842]  ? show_regs.cold+0x8/0x10
[27494.325845]  ? do_free_extent_accounting.cold+0xec/0x1f0 [btrfs]
[27494.325928]  ? __warn.cold+0xa7/0x101
[27494.325934]  ? do_free_extent_accounting.cold+0xec/0x1f0 [btrfs]
[27494.326010]  ? report_bug+0x114/0x160
[27494.326015]  ? irq_work_queue+0x2d/0x70
[27494.326021]  ? handle_bug+0x51/0xa0
[27494.326025]  ? exc_invalid_op+0x18/0x80
[27494.326028]  ? asm_exc_invalid_op+0x1b/0x20
[27494.326034]  ? do_free_extent_accounting.cold+0xec/0x1f0 [btrfs]
[27494.326109]  __btrfs_free_extent.isra.0+0x62a/0x900 [btrfs]
[27494.326181]  run_delayed_data_ref+0x6c/0x1d0 [btrfs]
[27494.326247]  btrfs_run_delayed_refs_for_head+0x30a/0x420 [btrfs]
[27494.326312]  __btrfs_run_delayed_refs+0x106/0x1b0 [btrfs]
[27494.326375]  btrfs_run_delayed_refs+0x3c/0xd0 [btrfs]
[27494.326437]  btrfs_commit_transaction+0x6a/0xc70 [btrfs]
[27494.326513]  transaction_kthread+0x167/0x1d0 [btrfs]
[27494.326584]  ? __pfx_transaction_kthread+0x10/0x10 [btrfs]
[27494.326653]  kthread+0xe4/0x110
[27494.326657]  ? __pfx_kthread+0x10/0x10
[27494.326660]  ret_from_fork+0x47/0x70
[27494.326665]  ? __pfx_kthread+0x10/0x10
[27494.326667]  ret_from_fork_asm+0x1a/0x30
[27494.326673]  </TASK>
[27494.326674] ---[ end trace 0000000000000000 ]---
[27494.326678] BTRFS info (device sdc1 state EA): dumping space info:
[27494.326681] BTRFS info (device sdc1 state EA): space_info DATA has 
928231587840 free, is not full
[27494.326684] BTRFS info (device sdc1 state EA): space_info 
total=35882286448640, used=34948418105344, pinned=5635772416, 
reserved=0, may_use=0, readonly=983040 zone_unusable=0
[27494.326688] BTRFS info (device sdc1 state EA): space_info METADATA 
has -6576521216 free, is full
[27494.326691] BTRFS info (device sdc1 state EA): space_info 
total=78383153152, used=77616775168, pinned=71188480, 
reserved=695123968, may_use=6576521216, readonly=65536 zone_unusable=0
[27494.326695] BTRFS info (device sdc1 state EA): space_info SYSTEM has 
5980160 free, is not full
[27494.326698] BTRFS info (device sdc1 state EA): space_info 
total=8388608, used=2408448, pinned=0, reserved=0, may_use=0, readonly=0 
zone_unusable=0
[27494.326701] BTRFS info (device sdc1 state EA): global_block_rsv: size 
536870912 reserved 536805376
[27494.326703] BTRFS info (device sdc1 state EA): trans_block_rsv: size 
0 reserved 0
[27494.326706] BTRFS info (device sdc1 state EA): chunk_block_rsv: size 
0 reserved 0
[27494.326708] BTRFS info (device sdc1 state EA): delayed_block_rsv: 
size 0 reserved 0
[27494.326710] BTRFS info (device sdc1 state EA): delayed_refs_rsv: size 
1727549669376 reserved 6039715840
[27494.326713] BTRFS: error (device sdc1 state EA) in 
do_free_extent_accounting:2968: errno=-28 No space left
[27494.443071] BTRFS error (device sdc1 state EA): failed to run delayed 
ref for logical 7329665056768 num_bytes 65536 type 184 action 2 ref_mod 
1: -28
[27494.603135] BTRFS: error (device sdc1 state EA) in 
btrfs_run_delayed_refs:2207: errno=-28 No space left




Tomasz Chmielewski