* Re: BTRFS discard crash: failed to run delayed ref for logical 15506102321152 num_bytes 16384 type 182 action 2 ref_mod 1: -2 6.11.2)
2026-04-11 3:35 BTRFS discard crash: failed to run delayed ref for logical 15506102321152 num_bytes 16384 type 182 action 2 ref_mod 1: -2 6.11.2) Marc MERLIN
@ 2026-04-11 4:47 ` Qu Wenruo
2026-04-11 12:04 ` Roman Mamedov
` (3 subsequent siblings)
4 siblings, 0 replies; 43+ messages in thread
From: Qu Wenruo @ 2026-04-11 4:47 UTC (permalink / raw)
To: Marc MERLIN, linux-btrfs, Boris Burkov, Josef Bacik, QuWenruo,
Filipe Manana
Cc: Chris Murphy, Zygo Blaxell, Roman Mamedov, Su Yue
在 2026/4/11 13:05, Marc MERLIN 写道:
> [Is there a more appropriate way to report FS corruption? Looks like
> Emails to just linux-btrfs@vger.kernel.org do not get seen amongst all
> the patches hiding a normal Email]
>
> Howdy,
>
> I had btfrs filesystem on top of raid5 with 5 spinning drives.
> I mistakenly enabled discard by mistake which caused a crash when the discard thread tried
> to run (no discard on those drives)
> Kernel 6.12
>
> I worked on recovery using gemini 3.0 pro, mounting read only is fine, but I need read write
> or will waste days (probably weeks) recreating this entire 20TB+ backup over the internet
>
> I'm not qualified to say if everything Gemini said was correct, but I think summary is:
> 1) discard can apparently kill a filesystem when it's hard drives below (it did for me)
> 2) -o skip_balance,usebackuproot didn't help
> 3) no way to mount after space cache has been cleared and block-group-tree is enabled
> 4) still no way to mount read write after removing block-group-tree
>
> It started with:
> [23345.326321] BTRFS: error (device dm-0 state A) in do_free_extent_accounting:2996: errno=-2 No such entry
> [23345.336394] BTRFS error (device dm-0 state EA): failed to run delayed ref for logical 15506102321152 num_bytes 16384 type 182 action 2 ref_mod 1: -2
> [23345.350299] BTRFS: error (device dm-0 state EA) in btrfs_run_delayed_refs:2215: errno=-2 No such entry
> [23345.360154] BTRFS warning (device dm-0 state EA):
>
> I ended up with:
>
> moremagic:~# mount -t btrfs -o rw,skip_balance,space_cache=v2,clear_cache /dev/mapper/crypt_bcache0 /mnt/btrfs_bigbackup
> BTRFS: device label DS6 devid 1 transid 296950 /dev/mapper/crypt_bcache0 (251:0) scanned by mount (6029)
> BTRFS info (device dm-0): first mount of filesystem a97dec85-a0d5-42ab-a0ef-e9b7479fbe43
> BTRFS info (device dm-0): using crc32c (crc32c-generic) checksum algorithm
> BTRFS warning (device dm-0): read-write for sector size 4096 with page size 16384 is experimental
> BTRFS info (device dm-0): bdev /dev/mapper/crypt_bcache0 errs: wr 0, rd 0, flush 0, corrupt 5074, gen 0
> ------------[ cut here ]------------
> BTRFS: Transaction aborted (error -2)
> WARNING: CPU: 3 PID: 6029 at fs/btrfs/extent-tree.c:2996 __btrfs_free_extent.isra.0+0x13a0/0x14a0 [btrfs]
> Modules linked in: dm_crypt dm_mod bcache raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xt_MASQUERADE ipt_REJECT nf_reject_ipv4 xt_tcpudp xt_conntrack xt_LOG nf_log_syslog nft_compat nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nf_tables rfcomm algif_hash algif_skcipher af_alg bnep cp210x brcmfmac_wcc binfmt_misc usbserial hci_uart brcmfmac btbcm vc4 snd_soc_hdmi_codec brcmutil bluetooth drm_display_helper cfg80211 cec drm_dma_helper rpi_hevc_dec ecdh_generic v4l2_mem2mem ecc snd_soc_core pisp_be videobuf2_dma_contig v3d videobuf2_memops videobuf2_v4l2 gpu_sched rfkill videodev drm_shmem_helper snd_compress snd_pcm_dmaengine snd_pcm videobuf2_common rp1_pio snd_timer snd drm_kms_helper mc raspberrypi_gpiomem rp1_fw sg sch_fq_codel ecryptfs fuse drm drm_panel_orientation_quirks backlight nfnetlink ip_tables x_tables raid1 aes_ce_blk aes_ce_cipher ghash_ce gf128mul libaes sha2_ce spidev sha256_arm64 sha1_ce raspberrypi_hwmon sha1_generic ahci i2c_brcmstb spi_bcm2835
> md_mod gpio_keys libahci pwm_fan rp1_adc libata rp1_mailbox nvmem_rmem uio_pdrv_genirq uio btrfs blake2b_generic xor xor_neon raid6_pq zram lz4_compress ipv6
> CPU: 3 UID: 0 PID: 6029 Comm: mount Not tainted 6.12.47+rpt-rpi-2712 #1 Debian 1:6.12.47-1+rpt1
> Hardware name: Raspberry Pi 5 Model B Rev 1.1 (DT)
> pstate: 60400009 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
> pc : __btrfs_free_extent.isra.0+0x13a0/0x14a0 [btrfs]
> lr : __btrfs_free_extent.isra.0+0x13a0/0x14a0 [btrfs]
> sp : ffffc000868bb680
> x29: ffffc000868bb720 x28: 0000000000000000 x27: 0000000000002f02
> x26: 000000000000007f x25: ffff8001de833aa0 x24: 0000000000004000
> x23: 0000000000000000 x22: ffff800102b64e70 x21: 0000000000004000
> x20: 00000e1a4bb88000 x19: 00000000fffffffe x18: 0000000000000000
> x17: 0000000000000000 x16: 0000000000000000 x15: 0000000000000000
> x14: 0000000000000000 x13: 0000000000000000 x12: 0000000000000000
> x11: 00000000000000c0 x10: 0000000000001a40 x9 : ffffd06fce4e06c0
> x8 : ffff80011f56e0a0 x7 : 000000042f72a7bd x6 : 0000000000000039
> x5 : 0000000000000001 x4 : 0000000000001ab0 x3 : 0000000000000804
> x2 : 0000000000000000 x1 : 0000000000000000 x0 : ffff80011f56c600
> Call trace:
> __btrfs_free_extent.isra.0+0x13a0/0x14a0 [btrfs]
> __btrfs_run_delayed_refs+0x508/0xec0 [btrfs]
> btrfs_run_delayed_refs+0x48/0x198 [btrfs]
> btrfs_commit_transaction+0x88/0xe20 [btrfs]
> btrfs_recover_relocation+0x55c/0x5d0 [btrfs]
> btrfs_start_pre_rw_mount+0x1d4/0x470 [btrfs]
> open_ctree+0x101c/0x13b8 [btrfs]
> btrfs_get_tree+0x5b4/0x800 [btrfs]
> vfs_get_tree+0x30/0x108
> fc_mount+0x20/0x68
> btrfs_get_tree+0x238/0x800 [btrfs]
> vfs_get_tree+0x30/0x108
> vfs_cmd_create+0x58/0xf8
> __arm64_sys_fsconfig+0x444/0x5b8
> invoke_syscall+0x50/0x120
> el0_svc_common.constprop.0+0x48/0xf0
> do_el0_svc+0x24/0x38
> el0_svc+0x30/0xf8
> el0t_64_sync_handler+0x120/0x130
> el0t_64_sync+0x190/0x198
> ---[ end trace 0000000000000000 ]---
> BTRFS: error (device dm-0 state A) in do_free_extent_accounting:2996: errno=-2 No such entry
> BTRFS error (device dm-0 state EA): failed to run delayed ref for logical 15506102321152 num_bytes 16384 type 182 action 2 ref_mod 1: -2
There is a missing extent item for a shared tree block, aka extent tree
corruption.
Please try skip_balance to see if the fs can be mounted, then cancel the
relocation.
Then re-run btrfs check so we do not have balance complicating the
situation.
> BTRFS: error (device dm-0 state EA) in btrfs_run_delayed_refs:2215: errno=-2 No such entry
> BTRFS warning (device dm-0 state EA): failed to recover relocation: -2
> BTRFS error (device dm-0 state EA): commit super ret -30
> BTRFS error (device dm-0 state EA): open_ctree failed: -2
>
>
> Gemini said
Please drop whatever LLM says, for most cases it makes no difference nor
sense.
I even believe it's making things worse.
>
> more details:
> backuproot didn't work (read write)
> I was forced to run
> btrfstune --convert-from-block-group-tree /dev/mapper/crypt_bcache0
Please do not do whatever writes to the fs until you know why you should
do that.
And in this case, this will only make things worse.
Now I do not even know if this is the original problem or something
introduced by your writes.
At least btrfs-progs can add some checks to prevent such operations when
there is a running balance.
I believe that's the only thing we can benefit from your report.
Next time, please do not do whatever crazy/stupid things unless *YOU*
know the reason.
Thanks,
Qu
> because
> When you ran btrfs check --clear-space-cache v2, the tool did exactly
> what it was supposed to do: it deleted the Free Space Tree and removed
> the FREE_SPACE_TREE flag from your superblock.
> The Conflict: Your 23TB array was formatted with the modern
> block-group-tree feature (which speeds up mounting).
> The Kernel Rule: The Btrfs kernel code explicitly dictates: If the Block
> Group Tree is enabled, the Free Space Tree MUST also be enabled. * The
> Crash: Because the FREE_SPACE_TREE flag is now missing, the kernel sees
> an "illegal" superblock state and throws a fatal -22 error, refusing to
> proceed to the mount options.
>
> This was vexing, hours lost removing the block group tree.
> and when it was finally finished,
> mount -t btrfs -o skip_balance /dev/mapper/crypt_bcache0 /mnt/btrfs_bigbackup/
> did run, but crashed as above
>
> Now doing a repair in case it can salvage things.
>
> Marc
^ permalink raw reply [flat|nested] 43+ messages in thread* Re: BTRFS discard crash: failed to run delayed ref for logical 15506102321152 num_bytes 16384 type 182 action 2 ref_mod 1: -2 6.11.2)
2026-04-11 3:35 BTRFS discard crash: failed to run delayed ref for logical 15506102321152 num_bytes 16384 type 182 action 2 ref_mod 1: -2 6.11.2) Marc MERLIN
2026-04-11 4:47 ` Qu Wenruo
@ 2026-04-11 12:04 ` Roman Mamedov
2026-04-11 16:22 ` Marc MERLIN
2026-04-12 1:57 ` Marc MERLIN
` (2 subsequent siblings)
4 siblings, 1 reply; 43+ messages in thread
From: Roman Mamedov @ 2026-04-11 12:04 UTC (permalink / raw)
To: Marc MERLIN; +Cc: linux-btrfs, Qu Wenruo
On Fri, 10 Apr 2026 20:35:33 -0700
Marc MERLIN <marc@merlins.org> wrote:
> I had btfrs filesystem on top of raid5 with 5 spinning drives.
> I mistakenly enabled discard by mistake which caused a crash when the discard thread tried
> to run (no discard on those drives)
> Kernel 6.12
>
> I worked on recovery using gemini 3.0 pro, mounting read only is fine, but I need read write
> or will waste days (probably weeks) recreating this entire 20TB+ backup over the internet
>
> I'm not qualified to say if everything Gemini said was correct, but I think summary is:
> 1) discard can apparently kill a filesystem when it's hard drives below (it did for me)
Although some SMR HDDs might support TRIM, if you are certain none of yours
do, then issuing the TRIM command does not alter on-disk data in any way, and
after a device returns ENOTSUPP once, the block layer usually knows to stop
trying, it will return "not supported" to the upper layer right away, not
contacting the drive about that anymore.
As such it is unlikely that discard itself caused the issue in your case.
There is also no discard in the presented traces, and it is unfortunate you
didn't capture the initial "crash". So it might be just a coincidence, or it
might be not. And like Mr. Qu, I am also skeptical of the AI fantasies in this
case.
Be aware of the write hole issue when running Btrfs on top of a multi-device
mdraid. In case of a system crash, some devices might have stripes written and
synced to disk, and others not. This is can easily lead Btrfs into the
infamous "parent transid verify failed" state, from which there's no good way
out.
--
With respect,
Roman
^ permalink raw reply [flat|nested] 43+ messages in thread* Re: BTRFS discard crash: failed to run delayed ref for logical 15506102321152 num_bytes 16384 type 182 action 2 ref_mod 1: -2 6.11.2)
2026-04-11 12:04 ` Roman Mamedov
@ 2026-04-11 16:22 ` Marc MERLIN
0 siblings, 0 replies; 43+ messages in thread
From: Marc MERLIN @ 2026-04-11 16:22 UTC (permalink / raw)
To: Roman Mamedov, Qu Wenruo
Cc: linux-btrfs, Boris Burkov, Josef Bacik, QuWenruo, Filipe Manana,
Chris Murphy, Zygo Blaxell, Su Yue
Thanks both for the answers.
> didn't capture the initial "crash". So it might be just a coincidence, or it
> might be not. And like Mr. Qu, I am also skeptical of the AI fantasies in this
> case.
>
> Be aware of the write hole issue when running Btrfs on top of a multi-device
> mdraid. In case of a system crash, some devices might have stripes written and
> synced to disk, and others not. This is can easily lead Btrfs into the
> infamous "parent transid verify failed" state, from which there's no good way
> out.
1) there was no system crash or power off that I can remember, at least
not recently.
2) I do have all the logs from the start, here they are: https://pastebin.com/7HmQwy3n
3) AI may have been wrong about linking me enabling trim to the crash
but they sure happened a few minutes apart. Could have been coincidence.
4) write hole: I do have md5 "Intent Bitmap : Internal" which indeed
prioritizes rebuild over fixing the write hole (mdadm can't do both,
sadly). I'm honestly sad that mdadm does not allow PPL (closing write
hole and intent bitmap for reasonable rebuild times)
5) the mdadm layer does not help, I would love to use built in btrfs
raid5 but last info I read still says it also has write hole or other
issues and can't really ever be production ready
5) RST is supposed to fix this but https://btrfs.readthedocs.io/en/latest/Status.html
says it's not ready, and why I asked about status recently, no answer
yet: https://yhbt.net/lore/linux-btrfs/adbgT-3VINfJNctk@merlins.org/#r
So raid5 and btrfs are still problematic :-/
On Sat, Apr 11, 2026 at 02:17:24PM +0930, Qu Wenruo wrote:
> Please try skip_balance to see if the fs can be mounted, then cancel the
> relocation.
I tried many mounts with skip_balance, they all still crashed.
You can find them all in https://pastebin.com/7HmQwy3n
> Then re-run btrfs check so we do not have balance complicating the
> situation.
My first one crashed due to OOM, I added 64GB swap and am trying again.
> > btrfstune --convert-from-block-group-tree /dev/mapper/crypt_bcache0
>
> Please do not do whatever writes to the fs until you know why you should do
> that.
> And in this case, this will only make things worse.
Sequence of mount commands:
mount -t btrfs -o ro,nologreplay,skip_balance,clear_cache /dev/mapper/crypt_bcache0 /mnt/btrfs_bigbackup
umount /mnt/btrfs_bigbackup
=> worked, but ro
mount -t btrfs -o ro,nologreplay,skip_balance,clear_cache /dev/mapper/crypt_bcache0 /mnt/btrfs_bigbackup/
mount -o remount,rw,skip_balance /mnt/btrfs_bigbackup/
umount /mnt/btrfs_bigbackup/
=> Could not remount with skip_balance
all of these failed and caused the mounts in https://pastebin.com/7HmQwy3n
mount -t btrfs -o nologreplay,skip_balance,clear_cache /dev/mapper/crypt_bcache0 /mnt/btrfs_bigbackup/
mount -t btrfs -o skip_balance,clear_cache /dev/mapper/crypt_bcache0 /mnt/btrfs_bigbackup/
mount -t btrfs -o skip_balance,usebackuproot /dev/mapper/crypt_bcache0 /mnt/btrfs_bigbackup/
mount -t btrfs -o skip_balance /dev/mapper/crypt_bcache0 /mnt/btrfs_bigbackup/
mount -t btrfs -o rw,skip_balance,space_cache=v2,clear_cache /dev/mapper/crypt_bcache0 /mnt/btrfs_bigbackup
Once I dropped the cache (clear_cache), I was forced to downgrade
with --convert-from-block-group-tree
> Now I do not even know if this is the original problem or something
> introduced by your writes.
Hopefully https://pastebin.com/7HmQwy3n shows the original issue
> Next time, please do not do whatever crazy/stupid things unless *YOU* know
> the reason.
I'm in the middle of a maintenance, I don't have a support contract with
you the few people who know who to read this, and to be honest I have
found pretty much no good debug info or guide on the net.
Even looking for the status of RST and how usable, it is, or not, I
found nothing on the official pages, and when I wrote on this list to
ask, I got 0 reply.
So I'm not saying it's great or smart to use an LLM, but if there is no
easily findable (or any) information on how to debug all those things
without reading/knowing the kernel code, what is the recommended path
for an end user?
Thanks,
Marc
--
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: BTRFS discard crash: failed to run delayed ref for logical 15506102321152 num_bytes 16384 type 182 action 2 ref_mod 1: -2 6.11.2)
2026-04-11 3:35 BTRFS discard crash: failed to run delayed ref for logical 15506102321152 num_bytes 16384 type 182 action 2 ref_mod 1: -2 6.11.2) Marc MERLIN
2026-04-11 4:47 ` Qu Wenruo
2026-04-11 12:04 ` Roman Mamedov
@ 2026-04-12 1:57 ` Marc MERLIN
2026-04-12 1:57 ` Marc MERLIN
2026-04-12 2:28 ` Marc MERLIN
2026-04-13 17:52 ` Simple quota unsafe? RIP: 0010:__btrfs_free_extent.isra.0+0xc41/0x1020 [btrfs] / do_free_extent_accounting:2999: errno=-2 No such entry Marc MERLIN
2026-04-17 3:43 ` BTRFS discard crash: failed to run delayed ref for logical 15506102321152 num_bytes 16384 type 182 action 2 ref_mod 1: -2 6.11.2) David Disseldorp
4 siblings, 2 replies; 43+ messages in thread
From: Marc MERLIN @ 2026-04-12 1:57 UTC (permalink / raw)
To: linux-btrfs, Boris Burkov, Josef Bacik, QuWenruo, Qu Wenruo,
Filipe Manana
Cc: Chris Murphy, Zygo Blaxell, Roman Mamedov, Su Yue, Su Yue
So, btrfs repair is weird. The "real one" just OOM's even if I give it
64GB of swap, because I guess it wants gigabytes of RAM in huge chunks
that can't be swapped.
moremagic:~# btrfs check --repair /dev/mapper/crypt_bcache0
enabling repair mode
WARNING:
Do not use --repair unless you are advised to do so by a developer
or an experienced user, and then only after having accepted that no
fsck can successfully repair all types of filesystem corruption. E.g.
some software or hardware bugs can fatally damage a volume.
The operation will start in 10 seconds.
Use Ctrl-C to stop it.
10 9 8 7 6 5 4 3 2 1
Starting repair.
Opening filesystem to check...
Checking filesystem on /dev/mapper/crypt_bcache0
UUID: a97dec85-a0d5-42ab-a0ef-e9b7479fbe43
[1/8] checking log skipped (none written)
[2/8] checking root items
Fixed 0 roots.
[3/8] checking extents
super bytes used 18659783561216 mismatches actual used 18659783544832
super bytes used 18659783544832 mismatches actual used 18659783593984
No device size related problem found
[4/8] checking free space cache
[5/8] checking fs roots
Killed
But this is strange lowmem is giving a totally different result, it may
just be entirely trashing my FS as I write this, but if it doesn't
succeed, I need to wipe it and start over anyway
moremagic:~# btrfs check --mode lowmem /dev/mapper/crypt_bcache0
Opening filesystem to check...
Checking filesystem on /dev/mapper/crypt_bcache0
UUID: a97dec85-a0d5-42ab-a0ef-e9b7479fbe43
[1/8] checking log skipped (none written)
[2/8] checking root items
[3/8] checking extents
ERROR: extent[16842752 168 4096] has unknown ref type: 172
ERROR: extent[16855040 168 4096] has unknown ref type: 172
ERROR: extent[1121296384 168 8192] has unknown ref type: 172
Gemnini said that's the simple quotas not supported in lowmem
moremagic:~# btrfstune --remove-simple-quota /dev/mapper/crypt_bcache0
bad eb member end: ptr 0x4000 start 15495212859392 member offset 16384 size 1
bad eb member end: ptr 0x4000 start 16568940527616 member offset 16384 size 1
bad eb member end: ptr 0x4000 start 16133001379840 member offset 16384 size 1
bad eb member end: ptr 0x4000 start 15495296155648 member offset 16384 size 1
bad eb member end: ptr 0x4000 start 15641227673600 member offset 16384 size 1
bad eb member end: ptr 0x4000 start 16027774648320 member offset 16384 size 1
bad eb member end: ptr 0x4000 start 16217827999744 member offset 16384 size 1
bad eb member end: ptr 0x4000 start 16217830113280 member offset 16384 size 1
bad eb member end: ptr 0x4000 start 15505949786112 member offset 16384 size 1
bad eb member end: ptr 0x4000 start 16357413355520 member offset 16384 size 1
bad eb member end: ptr 0x4000 start 15495414267904 member offset 16384 size 1
bad eb member end: ptr 0x4000 start 3133210902528 member offset 16384 size 1
bad eb member end: ptr 0x4000 start 16027775500288 member offset 16384 size 1
bad eb member end: ptr 0x4000 start 16217837060096 member offset 16384 size 1
bad eb member end: ptr 0x4000 start 15181688930304 member offset 16384 size 1
bad eb member end: ptr 0x4000 start 15181689208832 member offset 16384 size 1
bad eb member end: ptr 0x4000 start 16349905764352 member offset 16384 size 1
bad eb member end: ptr 0x4000 start 16349906010112 member offset 16384 size 1
So basically it looks like I'm kind of screwed unless I move the array
(remote, can't get to it right now) to a system with 64GB of RAM or whatnot.
Back to the original point, this is kind of sad
1) mdadm raid5 can't close the right hole while intent bitmaps are on
2) native btrfs raid5 has never been stable enough to be used in production
3) RST should eventually be, but nothing I read says it is today
4) btrfs check --repair still apparently requires at least as much RAM
as the filesystem size, which is "problematic"
5) --lowmem is out of date and not usable.
So, am I pretty much screwed and need to wipe, restart, and speed
probably weeks trying to resync all that data over the internet, or is
there a way out?
Thanks,
Marc
On Fri, Apr 10, 2026 at 08:35:33PM -0700, Marc MERLIN wrote:
> [Is there a more appropriate way to report FS corruption? Looks like
> Emails to just linux-btrfs@vger.kernel.org do not get seen amongst all
> the patches hiding a normal Email]
>
> Howdy,
>
> I had btfrs filesystem on top of raid5 with 5 spinning drives.
> I mistakenly enabled discard by mistake which caused a crash when the discard thread tried
> to run (no discard on those drives)
> Kernel 6.12
>
> I worked on recovery using gemini 3.0 pro, mounting read only is fine, but I need read write
> or will waste days (probably weeks) recreating this entire 20TB+ backup over the internet
>
> I'm not qualified to say if everything Gemini said was correct, but I think summary is:
> 1) discard can apparently kill a filesystem when it's hard drives below (it did for me)
> 2) -o skip_balance,usebackuproot didn't help
> 3) no way to mount after space cache has been cleared and block-group-tree is enabled
> 4) still no way to mount read write after removing block-group-tree
>
> It started with:
> [23345.326321] BTRFS: error (device dm-0 state A) in do_free_extent_accounting:2996: errno=-2 No such entry
> [23345.336394] BTRFS error (device dm-0 state EA): failed to run delayed ref for logical 15506102321152 num_bytes 16384 type 182 action 2 ref_mod 1: -2
> [23345.350299] BTRFS: error (device dm-0 state EA) in btrfs_run_delayed_refs:2215: errno=-2 No such entry
> [23345.360154] BTRFS warning (device dm-0 state EA):
>
> I ended up with:
>
> moremagic:~# mount -t btrfs -o rw,skip_balance,space_cache=v2,clear_cache /dev/mapper/crypt_bcache0 /mnt/btrfs_bigbackup
> BTRFS: device label DS6 devid 1 transid 296950 /dev/mapper/crypt_bcache0 (251:0) scanned by mount (6029)
> BTRFS info (device dm-0): first mount of filesystem a97dec85-a0d5-42ab-a0ef-e9b7479fbe43
> BTRFS info (device dm-0): using crc32c (crc32c-generic) checksum algorithm
> BTRFS warning (device dm-0): read-write for sector size 4096 with page size 16384 is experimental
> BTRFS info (device dm-0): bdev /dev/mapper/crypt_bcache0 errs: wr 0, rd 0, flush 0, corrupt 5074, gen 0
> ------------[ cut here ]------------
> BTRFS: Transaction aborted (error -2)
> WARNING: CPU: 3 PID: 6029 at fs/btrfs/extent-tree.c:2996 __btrfs_free_extent.isra.0+0x13a0/0x14a0 [btrfs]
> Modules linked in: dm_crypt dm_mod bcache raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xt_MASQUERADE ipt_REJECT nf_reject_ipv4 xt_tcpudp xt_conntrack xt_LOG nf_log_syslog nft_compat nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nf_tables rfcomm algif_hash algif_skcipher af_alg bnep cp210x brcmfmac_wcc binfmt_misc usbserial hci_uart brcmfmac btbcm vc4 snd_soc_hdmi_codec brcmutil bluetooth drm_display_helper cfg80211 cec drm_dma_helper rpi_hevc_dec ecdh_generic v4l2_mem2mem ecc snd_soc_core pisp_be videobuf2_dma_contig v3d videobuf2_memops videobuf2_v4l2 gpu_sched rfkill videodev drm_shmem_helper snd_compress snd_pcm_dmaengine snd_pcm videobuf2_common rp1_pio snd_timer snd drm_kms_helper mc raspberrypi_gpiomem rp1_fw sg sch_fq_codel ecryptfs fuse drm drm_panel_orientation_quirks backlight nfnetlink ip_tables x_tables raid1 aes_ce_blk aes_ce_cipher ghash_ce gf128mul libaes sha2_ce spidev sha256_arm64 sha1_ce raspberrypi_hwmon sha1_generic ahci i2c_brcmstb spi_bcm2835
> md_mod gpio_keys libahci pwm_fan rp1_adc libata rp1_mailbox nvmem_rmem uio_pdrv_genirq uio btrfs blake2b_generic xor xor_neon raid6_pq zram lz4_compress ipv6
> CPU: 3 UID: 0 PID: 6029 Comm: mount Not tainted 6.12.47+rpt-rpi-2712 #1 Debian 1:6.12.47-1+rpt1
> Hardware name: Raspberry Pi 5 Model B Rev 1.1 (DT)
> pstate: 60400009 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
> pc : __btrfs_free_extent.isra.0+0x13a0/0x14a0 [btrfs]
> lr : __btrfs_free_extent.isra.0+0x13a0/0x14a0 [btrfs]
> sp : ffffc000868bb680
> x29: ffffc000868bb720 x28: 0000000000000000 x27: 0000000000002f02
> x26: 000000000000007f x25: ffff8001de833aa0 x24: 0000000000004000
> x23: 0000000000000000 x22: ffff800102b64e70 x21: 0000000000004000
> x20: 00000e1a4bb88000 x19: 00000000fffffffe x18: 0000000000000000
> x17: 0000000000000000 x16: 0000000000000000 x15: 0000000000000000
> x14: 0000000000000000 x13: 0000000000000000 x12: 0000000000000000
> x11: 00000000000000c0 x10: 0000000000001a40 x9 : ffffd06fce4e06c0
> x8 : ffff80011f56e0a0 x7 : 000000042f72a7bd x6 : 0000000000000039
> x5 : 0000000000000001 x4 : 0000000000001ab0 x3 : 0000000000000804
> x2 : 0000000000000000 x1 : 0000000000000000 x0 : ffff80011f56c600
> Call trace:
> __btrfs_free_extent.isra.0+0x13a0/0x14a0 [btrfs]
> __btrfs_run_delayed_refs+0x508/0xec0 [btrfs]
> btrfs_run_delayed_refs+0x48/0x198 [btrfs]
> btrfs_commit_transaction+0x88/0xe20 [btrfs]
> btrfs_recover_relocation+0x55c/0x5d0 [btrfs]
> btrfs_start_pre_rw_mount+0x1d4/0x470 [btrfs]
> open_ctree+0x101c/0x13b8 [btrfs]
> btrfs_get_tree+0x5b4/0x800 [btrfs]
> vfs_get_tree+0x30/0x108
> fc_mount+0x20/0x68
> btrfs_get_tree+0x238/0x800 [btrfs]
> vfs_get_tree+0x30/0x108
> vfs_cmd_create+0x58/0xf8
> __arm64_sys_fsconfig+0x444/0x5b8
> invoke_syscall+0x50/0x120
> el0_svc_common.constprop.0+0x48/0xf0
> do_el0_svc+0x24/0x38
> el0_svc+0x30/0xf8
> el0t_64_sync_handler+0x120/0x130
> el0t_64_sync+0x190/0x198
> ---[ end trace 0000000000000000 ]---
> BTRFS: error (device dm-0 state A) in do_free_extent_accounting:2996: errno=-2 No such entry
> BTRFS error (device dm-0 state EA): failed to run delayed ref for logical 15506102321152 num_bytes 16384 type 182 action 2 ref_mod 1: -2
> BTRFS: error (device dm-0 state EA) in btrfs_run_delayed_refs:2215: errno=-2 No such entry
> BTRFS warning (device dm-0 state EA): failed to recover relocation: -2
> BTRFS error (device dm-0 state EA): commit super ret -30
> BTRFS error (device dm-0 state EA): open_ctree failed: -2
>
>
> Gemini said
>
> The Btrfs "Ghost" Accounting When you added discard=async to your fstab
> (or remounted with it), you told the Btrfs kernel module to start a specific
> background thread.
> Btrfs's Perspective: "The user told me to use async discard. I will now start a
> list of every extent we delete so I can 'trim' them later in the background."
> The Problem: Btrfs doesn't check if the underlying dm-crypt device actually
> supports discards before it starts its own internal accounting.
> The Result: Btrfs started tracking a massive list of "extents to be discarded"
> in its memory and metadata.
>
> 2. The "No Such Entry" (-2) Race Condition The crash didn't happen because a
> command hit a drive; it happened because of a logic race inside the kernel's
> Btrfs code:
> The Balance Thread: You were running a balance. This thread moves data from "Old
> Block A" to "New Block B."
> The Discard Thread: Because discard=async was on, the discard thread saw "Old
> Block A" get freed. It put "Old Block A" on its "to-do list."
> The Metadata Conflict: The balance thread finished moving the data and
> successfully deleted the reference to "Old Block A" from the extent tree.
> The Crash: A few milliseconds later, the async discard thread woke up and tried
> to "pin" or "process" the metadata for "Old Block A." It looked in the tree,
> found nothing (because the balance already deleted it), and threw an ENOENT
> (Error -2: No such entry).
> Btrfs panicked: "Wait, I was told to discard this block, but it doesn't exist in
> my records anymore! Something is inconsistent!" → Transaction Abort.
>
> more details:
> backuproot didn't work (read write)
> I was forced to run
> btrfstune --convert-from-block-group-tree /dev/mapper/crypt_bcache0
> because
> When you ran btrfs check --clear-space-cache v2, the tool did exactly
> what it was supposed to do: it deleted the Free Space Tree and removed
> the FREE_SPACE_TREE flag from your superblock.
> The Conflict: Your 23TB array was formatted with the modern
> block-group-tree feature (which speeds up mounting).
> The Kernel Rule: The Btrfs kernel code explicitly dictates: If the Block
> Group Tree is enabled, the Free Space Tree MUST also be enabled. * The
> Crash: Because the FREE_SPACE_TREE flag is now missing, the kernel sees
> an "illegal" superblock state and throws a fatal -22 error, refusing to
> proceed to the mount options.
>
> This was vexing, hours lost removing the block group tree.
> and when it was finally finished,
> mount -t btrfs -o skip_balance /dev/mapper/crypt_bcache0 /mnt/btrfs_bigbackup/
> did run, but crashed as above
>
> Now doing a repair in case it can salvage things.
>
> Marc
> --
> "A mouse is a device used to point at the xterm you want to type in" - A.S.R.
>
> Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
>
--
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
^ permalink raw reply [flat|nested] 43+ messages in thread* Re: BTRFS discard crash: failed to run delayed ref for logical 15506102321152 num_bytes 16384 type 182 action 2 ref_mod 1: -2 6.11.2)
2026-04-12 1:57 ` Marc MERLIN
@ 2026-04-12 1:57 ` Marc MERLIN
2026-04-12 2:28 ` Marc MERLIN
1 sibling, 0 replies; 43+ messages in thread
From: Marc MERLIN @ 2026-04-12 1:57 UTC (permalink / raw)
To: linux-btrfs, Boris Burkov, Josef Bacik, QuWenruo, Qu Wenruo,
Filipe Manana
Cc: Chris Murphy, Zygo Blaxell, Roman Mamedov, Su Yue, Su Yue
So, btrfs repair is weird. The "real one" just OOM's even if I give it
64GB of swap, because I guess it wants gigabytes of RAM in huge chunks
that can't be swapped.
moremagic:~# btrfs check --repair /dev/mapper/crypt_bcache0
enabling repair mode
WARNING:
Do not use --repair unless you are advised to do so by a developer
or an experienced user, and then only after having accepted that no
fsck can successfully repair all types of filesystem corruption. E.g.
some software or hardware bugs can fatally damage a volume.
The operation will start in 10 seconds.
Use Ctrl-C to stop it.
10 9 8 7 6 5 4 3 2 1
Starting repair.
Opening filesystem to check...
Checking filesystem on /dev/mapper/crypt_bcache0
UUID: a97dec85-a0d5-42ab-a0ef-e9b7479fbe43
[1/8] checking log skipped (none written)
[2/8] checking root items
Fixed 0 roots.
[3/8] checking extents
super bytes used 18659783561216 mismatches actual used 18659783544832
super bytes used 18659783544832 mismatches actual used 18659783593984
No device size related problem found
[4/8] checking free space cache
[5/8] checking fs roots
Killed
But this is strange lowmem is giving a totally different result, it may
just be entirely trashing my FS as I write this, but if it doesn't
succeed, I need to wipe it and start over anyway
moremagic:~# btrfs check --mode lowmem /dev/mapper/crypt_bcache0
Opening filesystem to check...
Checking filesystem on /dev/mapper/crypt_bcache0
UUID: a97dec85-a0d5-42ab-a0ef-e9b7479fbe43
[1/8] checking log skipped (none written)
[2/8] checking root items
[3/8] checking extents
ERROR: extent[16842752 168 4096] has unknown ref type: 172
ERROR: extent[16855040 168 4096] has unknown ref type: 172
ERROR: extent[1121296384 168 8192] has unknown ref type: 172
Gemnini said that's the simple quotas not supported in lowmem
moremagic:~# btrfstune --remove-simple-quota /dev/mapper/crypt_bcache0
bad eb member end: ptr 0x4000 start 15495212859392 member offset 16384 size 1
bad eb member end: ptr 0x4000 start 16568940527616 member offset 16384 size 1
bad eb member end: ptr 0x4000 start 16133001379840 member offset 16384 size 1
bad eb member end: ptr 0x4000 start 15495296155648 member offset 16384 size 1
bad eb member end: ptr 0x4000 start 15641227673600 member offset 16384 size 1
bad eb member end: ptr 0x4000 start 16027774648320 member offset 16384 size 1
bad eb member end: ptr 0x4000 start 16217827999744 member offset 16384 size 1
bad eb member end: ptr 0x4000 start 16217830113280 member offset 16384 size 1
bad eb member end: ptr 0x4000 start 15505949786112 member offset 16384 size 1
bad eb member end: ptr 0x4000 start 16357413355520 member offset 16384 size 1
bad eb member end: ptr 0x4000 start 15495414267904 member offset 16384 size 1
bad eb member end: ptr 0x4000 start 3133210902528 member offset 16384 size 1
bad eb member end: ptr 0x4000 start 16027775500288 member offset 16384 size 1
bad eb member end: ptr 0x4000 start 16217837060096 member offset 16384 size 1
bad eb member end: ptr 0x4000 start 15181688930304 member offset 16384 size 1
bad eb member end: ptr 0x4000 start 15181689208832 member offset 16384 size 1
bad eb member end: ptr 0x4000 start 16349905764352 member offset 16384 size 1
bad eb member end: ptr 0x4000 start 16349906010112 member offset 16384 size 1
So basically it looks like I'm kind of screwed unless I move the array
(remote, can't get to it right now) to a system with 64GB of RAM or whatnot.
Back to the original point, this is kind of sad
1) mdadm raid5 can't close the right hole while intent bitmaps are on
2) native btrfs raid5 has never been stable enough to be used in production
3) RST should eventually be, but nothing I read says it is today
4) btrfs check --repair still apparently requires at least as much RAM
as the filesystem size, which is "problematic"
5) --lowmem is out of date and not usable.
So, am I pretty much screwed and need to wipe, restart, and speed
probably weeks trying to resync all that data over the internet, or is
there a way out?
Thanks,
Marc
On Fri, Apr 10, 2026 at 08:35:33PM -0700, Marc MERLIN wrote:
> [Is there a more appropriate way to report FS corruption? Looks like
> Emails to just linux-btrfs@vger.kernel.org do not get seen amongst all
> the patches hiding a normal Email]
>
> Howdy,
>
> I had btfrs filesystem on top of raid5 with 5 spinning drives.
> I mistakenly enabled discard by mistake which caused a crash when the discard thread tried
> to run (no discard on those drives)
> Kernel 6.12
>
> I worked on recovery using gemini 3.0 pro, mounting read only is fine, but I need read write
> or will waste days (probably weeks) recreating this entire 20TB+ backup over the internet
>
> I'm not qualified to say if everything Gemini said was correct, but I think summary is:
> 1) discard can apparently kill a filesystem when it's hard drives below (it did for me)
> 2) -o skip_balance,usebackuproot didn't help
> 3) no way to mount after space cache has been cleared and block-group-tree is enabled
> 4) still no way to mount read write after removing block-group-tree
>
> It started with:
> [23345.326321] BTRFS: error (device dm-0 state A) in do_free_extent_accounting:2996: errno=-2 No such entry
> [23345.336394] BTRFS error (device dm-0 state EA): failed to run delayed ref for logical 15506102321152 num_bytes 16384 type 182 action 2 ref_mod 1: -2
> [23345.350299] BTRFS: error (device dm-0 state EA) in btrfs_run_delayed_refs:2215: errno=-2 No such entry
> [23345.360154] BTRFS warning (device dm-0 state EA):
>
> I ended up with:
>
> moremagic:~# mount -t btrfs -o rw,skip_balance,space_cache=v2,clear_cache /dev/mapper/crypt_bcache0 /mnt/btrfs_bigbackup
> BTRFS: device label DS6 devid 1 transid 296950 /dev/mapper/crypt_bcache0 (251:0) scanned by mount (6029)
> BTRFS info (device dm-0): first mount of filesystem a97dec85-a0d5-42ab-a0ef-e9b7479fbe43
> BTRFS info (device dm-0): using crc32c (crc32c-generic) checksum algorithm
> BTRFS warning (device dm-0): read-write for sector size 4096 with page size 16384 is experimental
> BTRFS info (device dm-0): bdev /dev/mapper/crypt_bcache0 errs: wr 0, rd 0, flush 0, corrupt 5074, gen 0
> ------------[ cut here ]------------
> BTRFS: Transaction aborted (error -2)
> WARNING: CPU: 3 PID: 6029 at fs/btrfs/extent-tree.c:2996 __btrfs_free_extent.isra.0+0x13a0/0x14a0 [btrfs]
> Modules linked in: dm_crypt dm_mod bcache raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xt_MASQUERADE ipt_REJECT nf_reject_ipv4 xt_tcpudp xt_conntrack xt_LOG nf_log_syslog nft_compat nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nf_tables rfcomm algif_hash algif_skcipher af_alg bnep cp210x brcmfmac_wcc binfmt_misc usbserial hci_uart brcmfmac btbcm vc4 snd_soc_hdmi_codec brcmutil bluetooth drm_display_helper cfg80211 cec drm_dma_helper rpi_hevc_dec ecdh_generic v4l2_mem2mem ecc snd_soc_core pisp_be videobuf2_dma_contig v3d videobuf2_memops videobuf2_v4l2 gpu_sched rfkill videodev drm_shmem_helper snd_compress snd_pcm_dmaengine snd_pcm videobuf2_common rp1_pio snd_timer snd drm_kms_helper mc raspberrypi_gpiomem rp1_fw sg sch_fq_codel ecryptfs fuse drm drm_panel_orientation_quirks backlight nfnetlink ip_tables x_tables raid1 aes_ce_blk aes_ce_cipher ghash_ce gf128mul libaes sha2_ce spidev sha256_arm64 sha1_ce raspberrypi_hwmon sha1_generic ahci i2c_brcmstb spi_bcm2835
> md_mod gpio_keys libahci pwm_fan rp1_adc libata rp1_mailbox nvmem_rmem uio_pdrv_genirq uio btrfs blake2b_generic xor xor_neon raid6_pq zram lz4_compress ipv6
> CPU: 3 UID: 0 PID: 6029 Comm: mount Not tainted 6.12.47+rpt-rpi-2712 #1 Debian 1:6.12.47-1+rpt1
> Hardware name: Raspberry Pi 5 Model B Rev 1.1 (DT)
> pstate: 60400009 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
> pc : __btrfs_free_extent.isra.0+0x13a0/0x14a0 [btrfs]
> lr : __btrfs_free_extent.isra.0+0x13a0/0x14a0 [btrfs]
> sp : ffffc000868bb680
> x29: ffffc000868bb720 x28: 0000000000000000 x27: 0000000000002f02
> x26: 000000000000007f x25: ffff8001de833aa0 x24: 0000000000004000
> x23: 0000000000000000 x22: ffff800102b64e70 x21: 0000000000004000
> x20: 00000e1a4bb88000 x19: 00000000fffffffe x18: 0000000000000000
> x17: 0000000000000000 x16: 0000000000000000 x15: 0000000000000000
> x14: 0000000000000000 x13: 0000000000000000 x12: 0000000000000000
> x11: 00000000000000c0 x10: 0000000000001a40 x9 : ffffd06fce4e06c0
> x8 : ffff80011f56e0a0 x7 : 000000042f72a7bd x6 : 0000000000000039
> x5 : 0000000000000001 x4 : 0000000000001ab0 x3 : 0000000000000804
> x2 : 0000000000000000 x1 : 0000000000000000 x0 : ffff80011f56c600
> Call trace:
> __btrfs_free_extent.isra.0+0x13a0/0x14a0 [btrfs]
> __btrfs_run_delayed_refs+0x508/0xec0 [btrfs]
> btrfs_run_delayed_refs+0x48/0x198 [btrfs]
> btrfs_commit_transaction+0x88/0xe20 [btrfs]
> btrfs_recover_relocation+0x55c/0x5d0 [btrfs]
> btrfs_start_pre_rw_mount+0x1d4/0x470 [btrfs]
> open_ctree+0x101c/0x13b8 [btrfs]
> btrfs_get_tree+0x5b4/0x800 [btrfs]
> vfs_get_tree+0x30/0x108
> fc_mount+0x20/0x68
> btrfs_get_tree+0x238/0x800 [btrfs]
> vfs_get_tree+0x30/0x108
> vfs_cmd_create+0x58/0xf8
> __arm64_sys_fsconfig+0x444/0x5b8
> invoke_syscall+0x50/0x120
> el0_svc_common.constprop.0+0x48/0xf0
> do_el0_svc+0x24/0x38
> el0_svc+0x30/0xf8
> el0t_64_sync_handler+0x120/0x130
> el0t_64_sync+0x190/0x198
> ---[ end trace 0000000000000000 ]---
> BTRFS: error (device dm-0 state A) in do_free_extent_accounting:2996: errno=-2 No such entry
> BTRFS error (device dm-0 state EA): failed to run delayed ref for logical 15506102321152 num_bytes 16384 type 182 action 2 ref_mod 1: -2
> BTRFS: error (device dm-0 state EA) in btrfs_run_delayed_refs:2215: errno=-2 No such entry
> BTRFS warning (device dm-0 state EA): failed to recover relocation: -2
> BTRFS error (device dm-0 state EA): commit super ret -30
> BTRFS error (device dm-0 state EA): open_ctree failed: -2
>
>
> Gemini said
>
> The Btrfs "Ghost" Accounting When you added discard=async to your fstab
> (or remounted with it), you told the Btrfs kernel module to start a specific
> background thread.
> Btrfs's Perspective: "The user told me to use async discard. I will now start a
> list of every extent we delete so I can 'trim' them later in the background."
> The Problem: Btrfs doesn't check if the underlying dm-crypt device actually
> supports discards before it starts its own internal accounting.
> The Result: Btrfs started tracking a massive list of "extents to be discarded"
> in its memory and metadata.
>
> 2. The "No Such Entry" (-2) Race Condition The crash didn't happen because a
> command hit a drive; it happened because of a logic race inside the kernel's
> Btrfs code:
> The Balance Thread: You were running a balance. This thread moves data from "Old
> Block A" to "New Block B."
> The Discard Thread: Because discard=async was on, the discard thread saw "Old
> Block A" get freed. It put "Old Block A" on its "to-do list."
> The Metadata Conflict: The balance thread finished moving the data and
> successfully deleted the reference to "Old Block A" from the extent tree.
> The Crash: A few milliseconds later, the async discard thread woke up and tried
> to "pin" or "process" the metadata for "Old Block A." It looked in the tree,
> found nothing (because the balance already deleted it), and threw an ENOENT
> (Error -2: No such entry).
> Btrfs panicked: "Wait, I was told to discard this block, but it doesn't exist in
> my records anymore! Something is inconsistent!" → Transaction Abort.
>
> more details:
> backuproot didn't work (read write)
> I was forced to run
> btrfstune --convert-from-block-group-tree /dev/mapper/crypt_bcache0
> because
> When you ran btrfs check --clear-space-cache v2, the tool did exactly
> what it was supposed to do: it deleted the Free Space Tree and removed
> the FREE_SPACE_TREE flag from your superblock.
> The Conflict: Your 23TB array was formatted with the modern
> block-group-tree feature (which speeds up mounting).
> The Kernel Rule: The Btrfs kernel code explicitly dictates: If the Block
> Group Tree is enabled, the Free Space Tree MUST also be enabled. * The
> Crash: Because the FREE_SPACE_TREE flag is now missing, the kernel sees
> an "illegal" superblock state and throws a fatal -22 error, refusing to
> proceed to the mount options.
>
> This was vexing, hours lost removing the block group tree.
> and when it was finally finished,
> mount -t btrfs -o skip_balance /dev/mapper/crypt_bcache0 /mnt/btrfs_bigbackup/
> did run, but crashed as above
>
> Now doing a repair in case it can salvage things.
>
> Marc
> --
> "A mouse is a device used to point at the xterm you want to type in" - A.S.R.
>
> Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
>
--
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
^ permalink raw reply [flat|nested] 43+ messages in thread* Re: BTRFS discard crash: failed to run delayed ref for logical 15506102321152 num_bytes 16384 type 182 action 2 ref_mod 1: -2 6.11.2)
2026-04-12 1:57 ` Marc MERLIN
2026-04-12 1:57 ` Marc MERLIN
@ 2026-04-12 2:28 ` Marc MERLIN
2026-04-12 2:28 ` Marc MERLIN
2026-04-12 17:38 ` Marc MERLIN
1 sibling, 2 replies; 43+ messages in thread
From: Marc MERLIN @ 2026-04-12 2:28 UTC (permalink / raw)
To: linux-btrfs, Boris Burkov, Josef Bacik, QuWenruo, Qu Wenruo,
Filipe Manana
Cc: Chris Murphy, Zygo Blaxell, Roman Mamedov, Su Yue, Su Yue
On Sat, Apr 11, 2026 at 06:57:33PM -0700, Marc MERLIN wrote:
> But this is strange lowmem is giving a totally different result, it may
> just be entirely trashing my FS as I write this, but if it doesn't
> succeed, I need to wipe it and start over anyway
> moremagic:~# btrfs check --mode lowmem /dev/mapper/crypt_bcache0
> Opening filesystem to check...
> Checking filesystem on /dev/mapper/crypt_bcache0
> UUID: a97dec85-a0d5-42ab-a0ef-e9b7479fbe43
> [1/8] checking log skipped (none written)
> [2/8] checking root items
> [3/8] checking extents
> ERROR: extent[16842752 168 4096] has unknown ref type: 172
> ERROR: extent[16855040 168 4096] has unknown ref type: 172
> ERROR: extent[1121296384 168 8192] has unknown ref type: 172
>
> Gemnini said that's the simple quotas not supported in lowmem
>
> moremagic:~# btrfstune --remove-simple-quota /dev/mapper/crypt_bcache0
> bad eb member end: ptr 0x4000 start 15495212859392 member offset 16384 size 1
> bad eb member end: ptr 0x4000 start 16568940527616 member offset 16384 size 1
> bad eb member end: ptr 0x4000 start 16133001379840 member offset 16384 size 1
I forgot to mention:
moremagic:~# btrfs --version
btrfs-progs v6.17.1
-EXPERIMENTAL -INJECT -STATIC +LZO +ZSTD +UDEV +FSVERITY +ZONED CRYPTO=builtin
And gemini could be completely wrong, but since I have no other source of help
while typing those commands, and I need that filesystem back up, or
ron the sameestoring from backup, this is what it says now
The error message bad eb member end: ptr 0x4000 start ... member offset
16384 size 1 is a smoking gun for an off-by-one bug in the tool's
bounds-checker:
0x4000 / 16384: This is exactly the default nodesize (16KB) for Btrfs.
Offset 16384, Size 1: The code is trying to read/write a 1-byte member
that starts at the very last byte of the buffer and ends at 16385.
The Conflict: Because the extent buffer is only 16384 bytes (0 to
16383), the strict validation in the btrfs-progs library (likely
check_eb_range in kernel-lib/extent_io.c) throws a warning and blocks
the operation.
Why this happened with Simple Quotas
The BTRFS_EXTENT_OWNER_REF_KEY (Type 172) that caused your lowmem check
to fail is relatively new (introduced in Kernel 6.7). The code for
--remove-simple-quota in btrfstune (v6.13+) has to iterate the extent
tree and delete these items.
(Speculation) There appears to be a bug in how the iterator handles the
"next item" pointer when a BTRFS_EXTENT_OWNER_REF_KEY is the last item
in a leaf, or when the leaf is being rebalanced after a deletion.
Suggestions/ideas welcome, including my previous question of is there
even really a way to do software raid5 on linux with btrfs without
getting those seemingly unfixable and unrepairable btrfs corruption
issues? (I say this because of that having been my experience over 11 years
of trying to do this and getting too many of these corruption problems,
almost all of which were unrepairable and required a very slow restore
process for terabytes of data)
Marc
--
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: BTRFS discard crash: failed to run delayed ref for logical 15506102321152 num_bytes 16384 type 182 action 2 ref_mod 1: -2 6.11.2)
2026-04-12 2:28 ` Marc MERLIN
@ 2026-04-12 2:28 ` Marc MERLIN
2026-04-12 17:38 ` Marc MERLIN
1 sibling, 0 replies; 43+ messages in thread
From: Marc MERLIN @ 2026-04-12 2:28 UTC (permalink / raw)
To: linux-btrfs, Boris Burkov, Josef Bacik, QuWenruo, Qu Wenruo,
Filipe Manana
Cc: Chris Murphy, Zygo Blaxell, Roman Mamedov, Su Yue, Su Yue
On Sat, Apr 11, 2026 at 06:57:33PM -0700, Marc MERLIN wrote:
> But this is strange lowmem is giving a totally different result, it may
> just be entirely trashing my FS as I write this, but if it doesn't
> succeed, I need to wipe it and start over anyway
> moremagic:~# btrfs check --mode lowmem /dev/mapper/crypt_bcache0
> Opening filesystem to check...
> Checking filesystem on /dev/mapper/crypt_bcache0
> UUID: a97dec85-a0d5-42ab-a0ef-e9b7479fbe43
> [1/8] checking log skipped (none written)
> [2/8] checking root items
> [3/8] checking extents
> ERROR: extent[16842752 168 4096] has unknown ref type: 172
> ERROR: extent[16855040 168 4096] has unknown ref type: 172
> ERROR: extent[1121296384 168 8192] has unknown ref type: 172
>
> Gemnini said that's the simple quotas not supported in lowmem
>
> moremagic:~# btrfstune --remove-simple-quota /dev/mapper/crypt_bcache0
> bad eb member end: ptr 0x4000 start 15495212859392 member offset 16384 size 1
> bad eb member end: ptr 0x4000 start 16568940527616 member offset 16384 size 1
> bad eb member end: ptr 0x4000 start 16133001379840 member offset 16384 size 1
I forgot to mention:
moremagic:~# btrfs --version
btrfs-progs v6.17.1
-EXPERIMENTAL -INJECT -STATIC +LZO +ZSTD +UDEV +FSVERITY +ZONED CRYPTO=builtin
And gemini could be completely wrong, but since I have no other source of help
while typing those commands, and I need that filesystem back up, or
ron the sameestoring from backup, this is what it says now
The error message bad eb member end: ptr 0x4000 start ... member offset
16384 size 1 is a smoking gun for an off-by-one bug in the tool's
bounds-checker:
0x4000 / 16384: This is exactly the default nodesize (16KB) for Btrfs.
Offset 16384, Size 1: The code is trying to read/write a 1-byte member
that starts at the very last byte of the buffer and ends at 16385.
The Conflict: Because the extent buffer is only 16384 bytes (0 to
16383), the strict validation in the btrfs-progs library (likely
check_eb_range in kernel-lib/extent_io.c) throws a warning and blocks
the operation.
Why this happened with Simple Quotas
The BTRFS_EXTENT_OWNER_REF_KEY (Type 172) that caused your lowmem check
to fail is relatively new (introduced in Kernel 6.7). The code for
--remove-simple-quota in btrfstune (v6.13+) has to iterate the extent
tree and delete these items.
(Speculation) There appears to be a bug in how the iterator handles the
"next item" pointer when a BTRFS_EXTENT_OWNER_REF_KEY is the last item
in a leaf, or when the leaf is being rebalanced after a deletion.
Suggestions/ideas welcome, including my previous question of is there
even really a way to do software raid5 on linux with btrfs without
getting those seemingly unfixable and unrepairable btrfs corruption
issues? (I say this because of that having been my experience over 11 years
of trying to do this and getting too many of these corruption problems,
almost all of which were unrepairable and required a very slow restore
process for terabytes of data)
Marc
--
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: BTRFS discard crash: failed to run delayed ref for logical 15506102321152 num_bytes 16384 type 182 action 2 ref_mod 1: -2 6.11.2)
2026-04-12 2:28 ` Marc MERLIN
2026-04-12 2:28 ` Marc MERLIN
@ 2026-04-12 17:38 ` Marc MERLIN
2026-04-12 17:38 ` Marc MERLIN
2026-04-12 20:21 ` Marc MERLIN
1 sibling, 2 replies; 43+ messages in thread
From: Marc MERLIN @ 2026-04-12 17:38 UTC (permalink / raw)
To: linux-btrfs, Boris Burkov, Josef Bacik, QuWenruo, Qu Wenruo,
Filipe Manana
Cc: Chris Murphy, Zygo Blaxell, Roman Mamedov, Su Yue, Su Yue
Last update, looks like there is no way out for a regular user, which is diappointing.
btrfs check eventually worked with a lot of swap and a trick Gemini gave me:
moremagic:~# echo -1000 > /proc/$(pgrep -x btrfs)/oom_score_adj
moremagic:~# sysctl vm.swappiness=100
At least it got that part right, I was able to run the whole check:
Opening filesystem to check...
Checking filesystem on /dev/mapper/crypt_bcache0
UUID: a97dec85-a0d5-42ab-a0ef-e9b7479fbe43
[1/8] checking log skipped (none written)
[1/7] checking root items (0:06:49 elapsed, 35112838 items checked)
Fixed 0 roots.
No device size related problem found (1:11:30 elapsed, 4288885 items checked)
[2/7] checking extents (1:11:31 elapsed, 4288885 items checked)
[3/7] checking free space cache (0:07:16 elapsed, 20918 items checked)
[4/7] checking fs roots (6:16:17 elapsed, 2835820 items checked)
[5/7] checking csums (without verifying data) (0:50:04 elapsed, 10400884 items checked)
[6/7] checking root refs (0:00:00 elapsed, 112 items checked)
[7/7] checking quota groups (0:08:19 elapsed)
found 18657895645184 bytes used, no error found
total csum bytes: 18153824804
total tree bytes: 70266978304
total fs tree bytes: 46633795584
total extent tree bytes: 3394486272
btree space waste bytes: 11583902998
file data blocks allocated: 43054973927424
referenced 23610773848064
So, not bad, but not good either since it shows nothing fixed.
And unsurprisingly, I'm back to square 1
As per my previous comments on btrfs and raid5, I think it's now my 5,
6th or 7th time in 12 years losing a filesystem (defined by it went bad
and I can't recover it myself). I was really hopeful that after all
these years, fixes and new code, it would be more solid but at least for
me, it hasn't been :-/
Before I wipe it and start over, I'm willing to try a few things you'd like
to try and fix this issue kernel and/or btrfs check side.
I can build a TOT btrfs-progs, but can't upgrade the kernel as it's a Pi5 with
indeed a page size of 16384 which hopefully isn't the source of all of this
(retiring power hungry 400W-ish intel systems to save on electricity bills)
moremagic:~# mount -o rw,skip_balance /dev/mapper/crypt_bcache0 /mnt/btrfs_bigbackup
usb 2-1: reset SuperSpeed USB device number 3 using xhci-hcd
BTRFS: device label DS6 devid 1 transid 296961 /dev/mapper/crypt_bcache0 (251:0) scanned by mount (304163)
BTRFS info (device dm-0): first mount of filesystem a97dec85-a0d5-42ab-a0ef-e9b7479fbe43
BTRFS info (device dm-0): using crc32c (crc32c-generic) checksum algorithm
BTRFS info (device dm-0): forcing free space tree for sector size 4096 with page size 16384
BTRFS warning (device dm-0): read-write for sector size 4096 with page size 16384 is experimental
BTRFS info (device dm-0): bdev /dev/mapper/crypt_bcache0 errs: wr 0, rd 0, flush 0, corrupt 5074, gen 0
------------[ cut here ]------------
BTRFS: Transaction aborted (error -2)
WARNING: CPU: 1 PID: 304163 at fs/btrfs/extent-tree.c:2996 __btrfs_free_extent.isra.0+0x13a0/0x14a0 [btrfs]
Modules linked in: dm_crypt dm_mod bcache raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xt_MASQUERADE ipt_REJECT nf_reject_ipv4 xt_tcpudp xt_conntrack xt_LOG nf_log_syslog nft_compat nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nf_tables rfcomm algif_hash algif_skcipher af_alg bnep cp210x brcmfmac_wcc binfmt_misc usbserial hci_uart brcmfmac btbcm vc4 snd_soc_hdmi_codec brcmutil bluetooth drm_display_helper cfg80211 cec drm_dma_helper rpi_hevc_dec ecdh_generic v4l2_mem2mem ecc snd_soc_core pisp_be videobuf2_dma_contig v3d videobuf2_memops videobuf2_v4l2 gpu_sched rfkill videodev drm_shmem_helper snd_compress snd_pcm_dmaengine snd_pcm videobuf2_common rp1_pio snd_timer snd drm_kms_helper mc raspberrypi_gpiomem rp1_fw sg sch_fq_codel ecryptfs fuse drm drm_panel_orientation_quirks backlight nfnetlink ip_tables x_tables raid1 aes_ce_blk aes_ce_cipher ghash_ce gf128mul libaes sha2_ce spidev sha256_arm64 sha1_ce raspberrypi_hwmon sha1_generic ahci i2c_brcmstb spi_bcm2835
md_mod gpio_keys libahci pwm_fan rp1_adc libata rp1_mailbox nvmem_rmem uio_pdrv_genirq uio btrfs blake2b_generic xor xor_neon raid6_pq zram lz4_compress ipv6
CPU: 1 UID: 0 PID: 304163 Comm: mount Tainted: G W 6.12.47+rpt-rpi-2712 #1 Debian 1:6.12.47-1+rpt1
Tainted: [W]=WARN
Hardware name: Raspberry Pi 5 Model B Rev 1.1 (DT)
pstate: 60400009 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
pc : __btrfs_free_extent.isra.0+0x13a0/0x14a0 [btrfs]
lr : __btrfs_free_extent.isra.0+0x13a0/0x14a0 [btrfs]
sp : ffffc00086a93680
x29: ffffc00086a93720 x28: 0000000000000000 x27: 0000000000002f02
x26: 000000000000007f x25: ffff8000157cec30 x24: 0000000000004000
x23: 0000000000000000 x22: ffff800102b671e0 x21: 0000000000004000
x20: 00000e1a4bb88000 x19: 00000000fffffffe x18: 0000000000000006
x17: ffff800100393640 x16: 0000000000000020 x15: 0000000000000002
x14: 0000000000000001 x13: 0000000020000000 x12: 0000000000000000
x11: ffffd06fcf9884f8 x10: 0000000000000ca9 x9 : ffffd06fcf1deb40
x8 : 0000000000017fe8 x7 : 00000000fffff000 x6 : 0000000000000002
x5 : 0000000000000400 x4 : 0000000000000800 x3 : 0000000000000000
x2 : 0000000000000000 x1 : 0000000000000000 x0 : ffff80011f76c600
Call trace:
__btrfs_free_extent.isra.0+0x13a0/0x14a0 [btrfs]
__btrfs_run_delayed_refs+0x508/0xec0 [btrfs]
btrfs_run_delayed_refs+0x48/0x198 [btrfs]
btrfs_commit_transaction+0x88/0xe20 [btrfs]
btrfs_recover_relocation+0x55c/0x5d0 [btrfs]
btrfs_start_pre_rw_mount+0x1d4/0x470 [btrfs]
open_ctree+0x101c/0x13b8 [btrfs]
btrfs_get_tree+0x5b4/0x800 [btrfs]
vfs_get_tree+0x30/0x108
fc_mount+0x20/0x68
btrfs_get_tree+0x238/0x800 [btrfs]
vfs_get_tree+0x30/0x108
vfs_cmd_create+0x58/0xf8
__arm64_sys_fsconfig+0x444/0x5b8
invoke_syscall+0x50/0x120
el0_svc_common.constprop.0+0x48/0xf0
do_el0_svc+0x24/0x38
el0_svc+0x30/0xf8
el0t_64_sync_handler+0x120/0x130
el0t_64_sync+0x190/0x198
---[ end trace 0000000000000000 ]---
BTRFS: error (device dm-0 state A) in do_free_extent_accounting:2996: errno=-2 No such entry
BTRFS error (device dm-0 state EA): failed to run delayed ref for logical 15506102321152 num_bytes 16384 type 182 action 2 ref_mod 1: -2
BTRFS: error (device dm-0 state EA) in btrfs_run_delayed_refs:2215: errno=-2 No such entry
BTRFS warning (device dm-0 state EA): failed to recover relocation: -2
BTRFS error (device dm-0 state EA): commit super ret -30
mount: /mnt/btrfs_bigbackup: fsconfig() failed: No such file or directory.
dmesg(1) may have more information after failed mount system call.
BTRFS error (device dm-0 state EA): open_ctree failed: -2
--
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
^ permalink raw reply [flat|nested] 43+ messages in thread* Re: BTRFS discard crash: failed to run delayed ref for logical 15506102321152 num_bytes 16384 type 182 action 2 ref_mod 1: -2 6.11.2)
2026-04-12 17:38 ` Marc MERLIN
@ 2026-04-12 17:38 ` Marc MERLIN
2026-04-12 20:21 ` Marc MERLIN
1 sibling, 0 replies; 43+ messages in thread
From: Marc MERLIN @ 2026-04-12 17:38 UTC (permalink / raw)
To: linux-btrfs, Boris Burkov, Josef Bacik, QuWenruo, Qu Wenruo,
Filipe Manana
Cc: Chris Murphy, Zygo Blaxell, Roman Mamedov, Su Yue, Su Yue
Last update, looks like there is no way out for a regular user, which is diappointing.
btrfs check eventually worked with a lot of swap and a trick Gemini gave me:
moremagic:~# echo -1000 > /proc/$(pgrep -x btrfs)/oom_score_adj
moremagic:~# sysctl vm.swappiness=100
At least it got that part right, I was able to run the whole check:
Opening filesystem to check...
Checking filesystem on /dev/mapper/crypt_bcache0
UUID: a97dec85-a0d5-42ab-a0ef-e9b7479fbe43
[1/8] checking log skipped (none written)
[1/7] checking root items (0:06:49 elapsed, 35112838 items checked)
Fixed 0 roots.
No device size related problem found (1:11:30 elapsed, 4288885 items checked)
[2/7] checking extents (1:11:31 elapsed, 4288885 items checked)
[3/7] checking free space cache (0:07:16 elapsed, 20918 items checked)
[4/7] checking fs roots (6:16:17 elapsed, 2835820 items checked)
[5/7] checking csums (without verifying data) (0:50:04 elapsed, 10400884 items checked)
[6/7] checking root refs (0:00:00 elapsed, 112 items checked)
[7/7] checking quota groups (0:08:19 elapsed)
found 18657895645184 bytes used, no error found
total csum bytes: 18153824804
total tree bytes: 70266978304
total fs tree bytes: 46633795584
total extent tree bytes: 3394486272
btree space waste bytes: 11583902998
file data blocks allocated: 43054973927424
referenced 23610773848064
So, not bad, but not good either since it shows nothing fixed.
And unsurprisingly, I'm back to square 1
As per my previous comments on btrfs and raid5, I think it's now my 5,
6th or 7th time in 12 years losing a filesystem (defined by it went bad
and I can't recover it myself). I was really hopeful that after all
these years, fixes and new code, it would be more solid but at least for
me, it hasn't been :-/
Before I wipe it and start over, I'm willing to try a few things you'd like
to try and fix this issue kernel and/or btrfs check side.
I can build a TOT btrfs-progs, but can't upgrade the kernel as it's a Pi5 with
indeed a page size of 16384 which hopefully isn't the source of all of this
(retiring power hungry 400W-ish intel systems to save on electricity bills)
moremagic:~# mount -o rw,skip_balance /dev/mapper/crypt_bcache0 /mnt/btrfs_bigbackup
usb 2-1: reset SuperSpeed USB device number 3 using xhci-hcd
BTRFS: device label DS6 devid 1 transid 296961 /dev/mapper/crypt_bcache0 (251:0) scanned by mount (304163)
BTRFS info (device dm-0): first mount of filesystem a97dec85-a0d5-42ab-a0ef-e9b7479fbe43
BTRFS info (device dm-0): using crc32c (crc32c-generic) checksum algorithm
BTRFS info (device dm-0): forcing free space tree for sector size 4096 with page size 16384
BTRFS warning (device dm-0): read-write for sector size 4096 with page size 16384 is experimental
BTRFS info (device dm-0): bdev /dev/mapper/crypt_bcache0 errs: wr 0, rd 0, flush 0, corrupt 5074, gen 0
------------[ cut here ]------------
BTRFS: Transaction aborted (error -2)
WARNING: CPU: 1 PID: 304163 at fs/btrfs/extent-tree.c:2996 __btrfs_free_extent.isra.0+0x13a0/0x14a0 [btrfs]
Modules linked in: dm_crypt dm_mod bcache raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xt_MASQUERADE ipt_REJECT nf_reject_ipv4 xt_tcpudp xt_conntrack xt_LOG nf_log_syslog nft_compat nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nf_tables rfcomm algif_hash algif_skcipher af_alg bnep cp210x brcmfmac_wcc binfmt_misc usbserial hci_uart brcmfmac btbcm vc4 snd_soc_hdmi_codec brcmutil bluetooth drm_display_helper cfg80211 cec drm_dma_helper rpi_hevc_dec ecdh_generic v4l2_mem2mem ecc snd_soc_core pisp_be videobuf2_dma_contig v3d videobuf2_memops videobuf2_v4l2 gpu_sched rfkill videodev drm_shmem_helper snd_compress snd_pcm_dmaengine snd_pcm videobuf2_common rp1_pio snd_timer snd drm_kms_helper mc raspberrypi_gpiomem rp1_fw sg sch_fq_codel ecryptfs fuse drm drm_panel_orientation_quirks backlight nfnetlink ip_tables x_tables raid1 aes_ce_blk aes_ce_cipher ghash_ce gf128mul libaes sha2_ce spidev sha256_arm64 sha1_ce raspberrypi_hwmon sha1_generic ahci i2c_brcmstb spi_bcm2835
md_mod gpio_keys libahci pwm_fan rp1_adc libata rp1_mailbox nvmem_rmem uio_pdrv_genirq uio btrfs blake2b_generic xor xor_neon raid6_pq zram lz4_compress ipv6
CPU: 1 UID: 0 PID: 304163 Comm: mount Tainted: G W 6.12.47+rpt-rpi-2712 #1 Debian 1:6.12.47-1+rpt1
Tainted: [W]=WARN
Hardware name: Raspberry Pi 5 Model B Rev 1.1 (DT)
pstate: 60400009 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
pc : __btrfs_free_extent.isra.0+0x13a0/0x14a0 [btrfs]
lr : __btrfs_free_extent.isra.0+0x13a0/0x14a0 [btrfs]
sp : ffffc00086a93680
x29: ffffc00086a93720 x28: 0000000000000000 x27: 0000000000002f02
x26: 000000000000007f x25: ffff8000157cec30 x24: 0000000000004000
x23: 0000000000000000 x22: ffff800102b671e0 x21: 0000000000004000
x20: 00000e1a4bb88000 x19: 00000000fffffffe x18: 0000000000000006
x17: ffff800100393640 x16: 0000000000000020 x15: 0000000000000002
x14: 0000000000000001 x13: 0000000020000000 x12: 0000000000000000
x11: ffffd06fcf9884f8 x10: 0000000000000ca9 x9 : ffffd06fcf1deb40
x8 : 0000000000017fe8 x7 : 00000000fffff000 x6 : 0000000000000002
x5 : 0000000000000400 x4 : 0000000000000800 x3 : 0000000000000000
x2 : 0000000000000000 x1 : 0000000000000000 x0 : ffff80011f76c600
Call trace:
__btrfs_free_extent.isra.0+0x13a0/0x14a0 [btrfs]
__btrfs_run_delayed_refs+0x508/0xec0 [btrfs]
btrfs_run_delayed_refs+0x48/0x198 [btrfs]
btrfs_commit_transaction+0x88/0xe20 [btrfs]
btrfs_recover_relocation+0x55c/0x5d0 [btrfs]
btrfs_start_pre_rw_mount+0x1d4/0x470 [btrfs]
open_ctree+0x101c/0x13b8 [btrfs]
btrfs_get_tree+0x5b4/0x800 [btrfs]
vfs_get_tree+0x30/0x108
fc_mount+0x20/0x68
btrfs_get_tree+0x238/0x800 [btrfs]
vfs_get_tree+0x30/0x108
vfs_cmd_create+0x58/0xf8
__arm64_sys_fsconfig+0x444/0x5b8
invoke_syscall+0x50/0x120
el0_svc_common.constprop.0+0x48/0xf0
do_el0_svc+0x24/0x38
el0_svc+0x30/0xf8
el0t_64_sync_handler+0x120/0x130
el0t_64_sync+0x190/0x198
---[ end trace 0000000000000000 ]---
BTRFS: error (device dm-0 state A) in do_free_extent_accounting:2996: errno=-2 No such entry
BTRFS error (device dm-0 state EA): failed to run delayed ref for logical 15506102321152 num_bytes 16384 type 182 action 2 ref_mod 1: -2
BTRFS: error (device dm-0 state EA) in btrfs_run_delayed_refs:2215: errno=-2 No such entry
BTRFS warning (device dm-0 state EA): failed to recover relocation: -2
BTRFS error (device dm-0 state EA): commit super ret -30
mount: /mnt/btrfs_bigbackup: fsconfig() failed: No such file or directory.
dmesg(1) may have more information after failed mount system call.
BTRFS error (device dm-0 state EA): open_ctree failed: -2
--
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
^ permalink raw reply [flat|nested] 43+ messages in thread* Re: BTRFS discard crash: failed to run delayed ref for logical 15506102321152 num_bytes 16384 type 182 action 2 ref_mod 1: -2 6.11.2)
2026-04-12 17:38 ` Marc MERLIN
2026-04-12 17:38 ` Marc MERLIN
@ 2026-04-12 20:21 ` Marc MERLIN
2026-04-12 20:21 ` Marc MERLIN
2026-04-13 2:14 ` Roman Mamedov
1 sibling, 2 replies; 43+ messages in thread
From: Marc MERLIN @ 2026-04-12 20:21 UTC (permalink / raw)
To: linux-btrfs, Boris Burkov, Josef Bacik, QuWenruo, Qu Wenruo,
Filipe Manana
Cc: Chris Murphy, Zygo Blaxell, Roman Mamedov, Su Yue, Su Yue
One last ditch effort from Gemini:
moremagic:~# btrfs rescue zero-log /dev/mapper/crypt_bcache0
Clearing log on /dev/mapper/crypt_bcache0, previous log_root 0, level 0
moremagic:~# mount -o rw,clear_cache,skip_balance /dev/mapper/crypt_bcache0 /mnt/btrfs_bigbackup
mount: /mnt/btrfs_bigbackup: fsconfig() failed: No such file or directory.
dmesg(1) may have more information after failed mount system call.
moremagic:~# mount -wno remount,skip_balance /mnt/btrfs_bigbackup/
^^^ works, filesystem is still there:
Read/write still failing :-/
Gemini says the following, whether it's right or wrong about the
details, it would be nice to have a way to btrfs --repair or force
cleanup/revert/delete on read-write mount to recover to a working state.
Coping 22TB off from r/o FS is pointless, it's an offsite backup, never mind
that I have nowhere to copy that too anyway, I really would like to
get it back to read/write if there isn't massive corruption, and there
does not seem to be.
Btrfs operates on a strict "clean up your mess first" policy.
Read-Only: The kernel simply reads the B-trees as they currently exist
on the disk. It completely ignores the Btrfs "to-do" list (the delayed
refs and the aborted relocation tree).
Read-Write: The kernel refuses to let you write new data until it
finishes the aborted relocation from when the system crashed days
ago. It tries to execute that "to-do" list, hits the corrupted Extent
Tree/Squota bug (ENOENT), and instantly panics and aborts the mount to
prevent further corruption.
Because there is no skip_relocation mount option in the Linux kernel,
you cannot force this filesystem to mount Read-Write without clearing
that broken tree. And because standard tools cannot clear a broken
relocation tree offline, the RW mount is effectively bricked.
moremagic:~# mount -wno remount,skip_balance /mnt/btrfs_bigbackup/
kernel: usb 2-1: reset SuperSpeed USB device number 3 using xhci-hcd
kernel: ------------[ cut here ]------------
kernel: BTRFS: Transaction aborted (error -2)
kernel: WARNING: CPU: 1 PID: 325244 at fs/btrfs/extent-tree.c:2996 __btrfs_free_extent.isra.0+0x13a0/0x14a0 [btrfs]
kernel: Modules linked in: dm_crypt dm_mod bcache raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xt_MASQUERADE ipt_REJECT nf_reject_ipv4 xt_tcpudp xt_conntrack xt_LOG nf_log_syslog nft_compat nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nf_tables rfcomm algif_hash algif_skcipher af_alg bnep cp210x brcmfmac_wcc binfmt_misc usbserial hci_uart brcmfmac btbcm vc4 snd_soc_hdmi_codec brcmutil bluetooth drm_display_helper cfg80211 cec drm_dma_helper rpi_hevc_dec ecdh_generic v4l2_mem2mem ecc snd_soc_core pisp_be videobuf2_dma_contig v3d videobuf2_memops videobuf2_v4l2 gpu_sched rfkill videodev drm_shmem_helper snd_compress snd_pcm_dmaengine snd_pcm videobuf2_common rp1_pio snd_timer snd drm_kms_helper mc raspberrypi_gpiomem rp1_fw sg sch_fq_codel ecryptfs fuse drm drm_panel_orientation_quirks backlight nfnetlink ip_tables x_tables raid1 aes_ce_blk aes_ce_cipher ghash_ce gf128mul libaes sha2_ce spidev sha256_arm64 sha1_ce raspberrypi_hwmon sha1_generic ahci i2c_brcmstb spi_bcm2835
kernel: md_mod gpio_keys libahci pwm_fan rp1_adc libata rp1_mailbox nvmem_rmem uio_pdrv_genirq uio btrfs blake2b_generic xor xor_neon raid6_pq zram lz4_compress ipv6
kernel: CPU: 1 UID: 0 PID: 325244 Comm: mount Tainted: G W 6.12.47+rpt-rpi-2712 #1 Debian 1:6.12.47-1+rpt1
kernel: Tainted: [W]=WARN
kernel: Hardware name: Raspberry Pi 5 Model B Rev 1.1 (DT)
kernel: pstate: 60400009 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
kernel: pc : __btrfs_free_extent.isra.0+0x13a0/0x14a0 [btrfs]
kernel: lr : __btrfs_free_extent.isra.0+0x13a0/0x14a0 [btrfs]
kernel: sp : ffffc0008a843820
kernel: x29: ffffc0008a8438c0 x28: 0000000000000000 x27: 0000000000002f02
kernel: x26: 000000000000007f x25: ffff80011fbbae60 x24: 0000000000004000
kernel: x23: 0000000000000000 x22: ffff8001033a9068 x21: 0000000000004000
kernel: x20: 00000e1a4bb88000 x19: 00000000fffffffe x18: 0000000000000000
kernel: x17: 0000000000000000 x16: 0000000000000000 x15: 0000000000000000
kernel: x14: 0000000000000000 x13: 0000000000000000 x12: 0000000000000000
kernel: x11: 00000000000000c0 x10: 0000000000001a40 x9 : ffffd06fce4e06c0
kernel: x8 : ffff80011e5a9ea0 x7 : ffffc0008293bc30 x6 : 0000000000000043
kernel: x5 : 0000000000000001 x4 : 0000000000001ab0 x3 : 0000000000000804
kernel: x2 : 0000000000000000 x1 : 0000000000000000 x0 : ffff80011e5a8400
kernel: Call trace:
kernel: __btrfs_free_extent.isra.0+0x13a0/0x14a0 [btrfs]
kernel: __btrfs_run_delayed_refs+0x508/0xec0 [btrfs]
kernel: btrfs_run_delayed_refs+0x48/0x198 [btrfs]
kernel: btrfs_commit_transaction+0x88/0xe20 [btrfs]
kernel: btrfs_recover_relocation+0x55c/0x5d0 [btrfs]
kernel: btrfs_start_pre_rw_mount+0x1d4/0x470 [btrfs]
kernel: btrfs_reconfigure+0x198/0x6d8 [btrfs]
kernel: reconfigure_super+0xc8/0x260
kernel: __arm64_sys_fsconfig+0x510/0x5b8
kernel: invoke_syscall+0x50/0x120
kernel: el0_svc_common.constprop.0+0x48/0xf0
kernel: do_el0_svc+0x24/0x38
kernel: el0_svc+0x30/0xf8
kernel: el0t_64_sync_handler+0x120/0x130
kernel: el0t_64_sync+0x190/0x198
kernel: ---[ end trace 0000000000000000 ]---
kernel: BTRFS: error (device dm-0 state MA) in do_free_extent_accounting:2996: errno=-2 No such entry
kernel: BTRFS error (device dm-0 state EMA): failed to run delayed ref for logical 15506102321152 num_bytes 16384 type 182 action 2 ref_mod 1: -2
kernel: BTRFS: error (device dm-0 state EMA) in btrfs_run_delayed_refs:2215: errno=-2 No such entry
kernel: BTRFS warning (device dm-0 state EMA): failed to recover relocation: -2
mount: /mnt/btrfs_bigbackup: fsconfig() failed: No such file or directory.
dmesg(1) may have more information after failed mount system call.
--
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
^ permalink raw reply [flat|nested] 43+ messages in thread* Re: BTRFS discard crash: failed to run delayed ref for logical 15506102321152 num_bytes 16384 type 182 action 2 ref_mod 1: -2 6.11.2)
2026-04-12 20:21 ` Marc MERLIN
@ 2026-04-12 20:21 ` Marc MERLIN
2026-04-13 2:14 ` Roman Mamedov
1 sibling, 0 replies; 43+ messages in thread
From: Marc MERLIN @ 2026-04-12 20:21 UTC (permalink / raw)
To: linux-btrfs, Boris Burkov, Josef Bacik, QuWenruo, Qu Wenruo,
Filipe Manana
Cc: Chris Murphy, Zygo Blaxell, Roman Mamedov, Su Yue, Su Yue
One last ditch effort from Gemini:
moremagic:~# btrfs rescue zero-log /dev/mapper/crypt_bcache0
Clearing log on /dev/mapper/crypt_bcache0, previous log_root 0, level 0
moremagic:~# mount -o rw,clear_cache,skip_balance /dev/mapper/crypt_bcache0 /mnt/btrfs_bigbackup
mount: /mnt/btrfs_bigbackup: fsconfig() failed: No such file or directory.
dmesg(1) may have more information after failed mount system call.
moremagic:~# mount -wno remount,skip_balance /mnt/btrfs_bigbackup/
^^^ works, filesystem is still there:
Read/write still failing :-/
Gemini says the following, whether it's right or wrong about the
details, it would be nice to have a way to btrfs --repair or force
cleanup/revert/delete on read-write mount to recover to a working state.
Coping 22TB off from r/o FS is pointless, it's an offsite backup, never mind
that I have nowhere to copy that too anyway, I really would like to
get it back to read/write if there isn't massive corruption, and there
does not seem to be.
Btrfs operates on a strict "clean up your mess first" policy.
Read-Only: The kernel simply reads the B-trees as they currently exist
on the disk. It completely ignores the Btrfs "to-do" list (the delayed
refs and the aborted relocation tree).
Read-Write: The kernel refuses to let you write new data until it
finishes the aborted relocation from when the system crashed days
ago. It tries to execute that "to-do" list, hits the corrupted Extent
Tree/Squota bug (ENOENT), and instantly panics and aborts the mount to
prevent further corruption.
Because there is no skip_relocation mount option in the Linux kernel,
you cannot force this filesystem to mount Read-Write without clearing
that broken tree. And because standard tools cannot clear a broken
relocation tree offline, the RW mount is effectively bricked.
moremagic:~# mount -wno remount,skip_balance /mnt/btrfs_bigbackup/
kernel: usb 2-1: reset SuperSpeed USB device number 3 using xhci-hcd
kernel: ------------[ cut here ]------------
kernel: BTRFS: Transaction aborted (error -2)
kernel: WARNING: CPU: 1 PID: 325244 at fs/btrfs/extent-tree.c:2996 __btrfs_free_extent.isra.0+0x13a0/0x14a0 [btrfs]
kernel: Modules linked in: dm_crypt dm_mod bcache raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xt_MASQUERADE ipt_REJECT nf_reject_ipv4 xt_tcpudp xt_conntrack xt_LOG nf_log_syslog nft_compat nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nf_tables rfcomm algif_hash algif_skcipher af_alg bnep cp210x brcmfmac_wcc binfmt_misc usbserial hci_uart brcmfmac btbcm vc4 snd_soc_hdmi_codec brcmutil bluetooth drm_display_helper cfg80211 cec drm_dma_helper rpi_hevc_dec ecdh_generic v4l2_mem2mem ecc snd_soc_core pisp_be videobuf2_dma_contig v3d videobuf2_memops videobuf2_v4l2 gpu_sched rfkill videodev drm_shmem_helper snd_compress snd_pcm_dmaengine snd_pcm videobuf2_common rp1_pio snd_timer snd drm_kms_helper mc raspberrypi_gpiomem rp1_fw sg sch_fq_codel ecryptfs fuse drm drm_panel_orientation_quirks backlight nfnetlink ip_tables x_tables raid1 aes_ce_blk aes_ce_cipher ghash_ce gf128mul libaes sha2_ce spidev sha256_arm64 sha1_ce raspberrypi_hwmon sha1_generic ahci i2c_brcmstb spi_bcm2835
kernel: md_mod gpio_keys libahci pwm_fan rp1_adc libata rp1_mailbox nvmem_rmem uio_pdrv_genirq uio btrfs blake2b_generic xor xor_neon raid6_pq zram lz4_compress ipv6
kernel: CPU: 1 UID: 0 PID: 325244 Comm: mount Tainted: G W 6.12.47+rpt-rpi-2712 #1 Debian 1:6.12.47-1+rpt1
kernel: Tainted: [W]=WARN
kernel: Hardware name: Raspberry Pi 5 Model B Rev 1.1 (DT)
kernel: pstate: 60400009 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
kernel: pc : __btrfs_free_extent.isra.0+0x13a0/0x14a0 [btrfs]
kernel: lr : __btrfs_free_extent.isra.0+0x13a0/0x14a0 [btrfs]
kernel: sp : ffffc0008a843820
kernel: x29: ffffc0008a8438c0 x28: 0000000000000000 x27: 0000000000002f02
kernel: x26: 000000000000007f x25: ffff80011fbbae60 x24: 0000000000004000
kernel: x23: 0000000000000000 x22: ffff8001033a9068 x21: 0000000000004000
kernel: x20: 00000e1a4bb88000 x19: 00000000fffffffe x18: 0000000000000000
kernel: x17: 0000000000000000 x16: 0000000000000000 x15: 0000000000000000
kernel: x14: 0000000000000000 x13: 0000000000000000 x12: 0000000000000000
kernel: x11: 00000000000000c0 x10: 0000000000001a40 x9 : ffffd06fce4e06c0
kernel: x8 : ffff80011e5a9ea0 x7 : ffffc0008293bc30 x6 : 0000000000000043
kernel: x5 : 0000000000000001 x4 : 0000000000001ab0 x3 : 0000000000000804
kernel: x2 : 0000000000000000 x1 : 0000000000000000 x0 : ffff80011e5a8400
kernel: Call trace:
kernel: __btrfs_free_extent.isra.0+0x13a0/0x14a0 [btrfs]
kernel: __btrfs_run_delayed_refs+0x508/0xec0 [btrfs]
kernel: btrfs_run_delayed_refs+0x48/0x198 [btrfs]
kernel: btrfs_commit_transaction+0x88/0xe20 [btrfs]
kernel: btrfs_recover_relocation+0x55c/0x5d0 [btrfs]
kernel: btrfs_start_pre_rw_mount+0x1d4/0x470 [btrfs]
kernel: btrfs_reconfigure+0x198/0x6d8 [btrfs]
kernel: reconfigure_super+0xc8/0x260
kernel: __arm64_sys_fsconfig+0x510/0x5b8
kernel: invoke_syscall+0x50/0x120
kernel: el0_svc_common.constprop.0+0x48/0xf0
kernel: do_el0_svc+0x24/0x38
kernel: el0_svc+0x30/0xf8
kernel: el0t_64_sync_handler+0x120/0x130
kernel: el0t_64_sync+0x190/0x198
kernel: ---[ end trace 0000000000000000 ]---
kernel: BTRFS: error (device dm-0 state MA) in do_free_extent_accounting:2996: errno=-2 No such entry
kernel: BTRFS error (device dm-0 state EMA): failed to run delayed ref for logical 15506102321152 num_bytes 16384 type 182 action 2 ref_mod 1: -2
kernel: BTRFS: error (device dm-0 state EMA) in btrfs_run_delayed_refs:2215: errno=-2 No such entry
kernel: BTRFS warning (device dm-0 state EMA): failed to recover relocation: -2
mount: /mnt/btrfs_bigbackup: fsconfig() failed: No such file or directory.
dmesg(1) may have more information after failed mount system call.
--
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
^ permalink raw reply [flat|nested] 43+ messages in thread* Re: BTRFS discard crash: failed to run delayed ref for logical 15506102321152 num_bytes 16384 type 182 action 2 ref_mod 1: -2 6.11.2)
2026-04-12 20:21 ` Marc MERLIN
2026-04-12 20:21 ` Marc MERLIN
@ 2026-04-13 2:14 ` Roman Mamedov
2026-04-13 2:34 ` Marc MERLIN
1 sibling, 1 reply; 43+ messages in thread
From: Roman Mamedov @ 2026-04-13 2:14 UTC (permalink / raw)
To: Marc MERLIN; +Cc: linux-btrfs, Qu Wenruo
On Sun, 12 Apr 2026 13:21:46 -0700
Marc MERLIN <marc@merlins.org> wrote:
> One last ditch effort from Gemini:
I think you'd better not put LLM usage in the forefront of your messages :)
Let alone copy and paste entire unedited paragraphs from it.
It acts as a red herring to many people, and those who would otherwise read
your posts and try to answer, will just glance over entirely.
> moremagic:~# btrfs rescue zero-log /dev/mapper/crypt_bcache0
> Clearing log on /dev/mapper/crypt_bcache0, previous log_root 0, level 0
> moremagic:~# mount -o rw,clear_cache,skip_balance /dev/mapper/crypt_bcache0 /mnt/btrfs_bigbackup
> mount: /mnt/btrfs_bigbackup: fsconfig() failed: No such file or directory.
> dmesg(1) may have more information after failed mount system call.
> moremagic:~# mount -wno remount,skip_balance /mnt/btrfs_bigbackup/
> ^^^ works, filesystem is still there:
>
> Read/write still failing :-/
Nothing wrong with just "this is what I tried and it didn't help", it's not as
important that LLM was the source of the ideas to try.
> Gemini says the following
Everyone has access to Gemini, and if Btrfs developers would be interested in
what it has to say about their filesystem, they could go and ask it directly :)
Secondly, I feel when others DO answer and something doesn't look right, you
may start your objection with "But Gemini said..."
And that's a whole another story, arguing with a person wielding an LLM and
trusting it more than a person in front of them. Many have been there by now
I'm sure. :)
> details, it would be nice to have a way to btrfs --repair or force
> cleanup/revert/delete on read-write mount to recover to a working state.
> Coping 22TB off from r/o FS is pointless, it's an offsite backup, never mind
> that I have nowhere to copy that too anyway, I really would like to
> get it back to read/write if there isn't massive corruption, and there
> does not seem to be.
>
> Btrfs operates on a strict "clean up your mess first" policy.
> Read-Only: The kernel simply reads the B-trees as they currently exist
> on the disk. It completely ignores the Btrfs "to-do" list (the delayed
> refs and the aborted relocation tree).
> Read-Write: The kernel refuses to let you write new data until it
> finishes the aborted relocation from when the system crashed days
> ago. It tries to execute that "to-do" list, hits the corrupted Extent
> Tree/Squota bug (ENOENT), and instantly panics and aborts the mount to
> prevent further corruption.
>
> Because there is no skip_relocation mount option in the Linux kernel,
> you cannot force this filesystem to mount Read-Write without clearing
> that broken tree. And because standard tools cannot clear a broken
> relocation tree offline, the RW mount is effectively bricked.
--
With respect,
Roman
^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: BTRFS discard crash: failed to run delayed ref for logical 15506102321152 num_bytes 16384 type 182 action 2 ref_mod 1: -2 6.11.2)
2026-04-13 2:14 ` Roman Mamedov
@ 2026-04-13 2:34 ` Marc MERLIN
2026-04-13 2:34 ` Marc MERLIN
0 siblings, 1 reply; 43+ messages in thread
From: Marc MERLIN @ 2026-04-13 2:34 UTC (permalink / raw)
To: Roman Mamedov; +Cc: linux-btrfs, Qu Wenruo
On Mon, Apr 13, 2026 at 07:14:19AM +0500, Roman Mamedov wrote:
> I think you'd better not put LLM usage in the forefront of your messages :)
I'm just being honest about how I got the commands, due to lack of
better online source I was able to find.
> > Gemini says the following
>
> Everyone has access to Gemini, and if Btrfs developers would be interested in
> what it has to say about their filesystem, they could go and ask it directly :)
I don't quite agree. If google search (before) and gemini (now) are
giving incorrect and potentially harmful answers, it's actually
important to know.
I'm a developer myself (for much simpler userspace code), and if I don't
have sufficient docs for my software and users end up finding and doing
the wrong things from google searches or LLM suggestions, I need and
want to know.
> And that's a whole another story, arguing with a person wielding an LLM and
> trusting it more than a person in front of them. Many have been there by now
> I'm sure. :)
I absolutely do not trust an LLM over anyone here or any actual
published doc I can find.
I've been stuck for 3 days with a broken filesystem I need to fix,
cannot find any docs that help (but there may not be any), am providing
as many ample logs and commands on what I'm doing and the kernel errors
I'm getting, as well as the workflow that is getting me from A to B to
C, and why I'm going that route, even I'm entirely wrong in doing so.
I do not expect anyone here to answer every few hours or daily or during
the weekend, or at all sicne this list is technically not a support
list, but then again I'm not sure there is one (I asked, got no answer
on that topic), so I can't wait days or weeks for an answer that may
never come.
So given that, LLM is worst option I have, but it's also the only one left.
Some steps like btrfs repair which I had to try 3 times at up to half a
day of runtime each time, are also good that I already took them and
finally got it to finish (and sadly fix nothing)
Still, even if my FS is hosed and no one can or has time to help, I'm
still spending the time to share what I found and hit, in case it can
help fix future kernels or versions of check --repair.
Since it's not safe for me to stay very long with my main backup mostly
dead/unusable, I'll wait another day or so in case any one here would
like to me to give more info/try things before I wipe, and if not, I'll
sadly have to do that and suck up the likely multi week data resync
Marc
--
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: BTRFS discard crash: failed to run delayed ref for logical 15506102321152 num_bytes 16384 type 182 action 2 ref_mod 1: -2 6.11.2)
2026-04-13 2:34 ` Marc MERLIN
@ 2026-04-13 2:34 ` Marc MERLIN
0 siblings, 0 replies; 43+ messages in thread
From: Marc MERLIN @ 2026-04-13 2:34 UTC (permalink / raw)
To: Roman Mamedov; +Cc: linux-btrfs, Qu Wenruo
On Mon, Apr 13, 2026 at 07:14:19AM +0500, Roman Mamedov wrote:
> I think you'd better not put LLM usage in the forefront of your messages :)
I'm just being honest about how I got the commands, due to lack of
better online source I was able to find.
> > Gemini says the following
>
> Everyone has access to Gemini, and if Btrfs developers would be interested in
> what it has to say about their filesystem, they could go and ask it directly :)
I don't quite agree. If google search (before) and gemini (now) are
giving incorrect and potentially harmful answers, it's actually
important to know.
I'm a developer myself (for much simpler userspace code), and if I don't
have sufficient docs for my software and users end up finding and doing
the wrong things from google searches or LLM suggestions, I need and
want to know.
> And that's a whole another story, arguing with a person wielding an LLM and
> trusting it more than a person in front of them. Many have been there by now
> I'm sure. :)
I absolutely do not trust an LLM over anyone here or any actual
published doc I can find.
I've been stuck for 3 days with a broken filesystem I need to fix,
cannot find any docs that help (but there may not be any), am providing
as many ample logs and commands on what I'm doing and the kernel errors
I'm getting, as well as the workflow that is getting me from A to B to
C, and why I'm going that route, even I'm entirely wrong in doing so.
I do not expect anyone here to answer every few hours or daily or during
the weekend, or at all sicne this list is technically not a support
list, but then again I'm not sure there is one (I asked, got no answer
on that topic), so I can't wait days or weeks for an answer that may
never come.
So given that, LLM is worst option I have, but it's also the only one left.
Some steps like btrfs repair which I had to try 3 times at up to half a
day of runtime each time, are also good that I already took them and
finally got it to finish (and sadly fix nothing)
Still, even if my FS is hosed and no one can or has time to help, I'm
still spending the time to share what I found and hit, in case it can
help fix future kernels or versions of check --repair.
Since it's not safe for me to stay very long with my main backup mostly
dead/unusable, I'll wait another day or so in case any one here would
like to me to give more info/try things before I wipe, and if not, I'll
sadly have to do that and suck up the likely multi week data resync
Marc
--
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
^ permalink raw reply [flat|nested] 43+ messages in thread
* Simple quota unsafe? RIP: 0010:__btrfs_free_extent.isra.0+0xc41/0x1020 [btrfs] / do_free_extent_accounting:2999: errno=-2 No such entry
2026-04-11 3:35 BTRFS discard crash: failed to run delayed ref for logical 15506102321152 num_bytes 16384 type 182 action 2 ref_mod 1: -2 6.11.2) Marc MERLIN
` (2 preceding siblings ...)
2026-04-12 1:57 ` Marc MERLIN
@ 2026-04-13 17:52 ` Marc MERLIN
2026-04-13 17:52 ` Marc MERLIN
2026-04-13 18:47 ` Boris Burkov
2026-04-17 3:43 ` BTRFS discard crash: failed to run delayed ref for logical 15506102321152 num_bytes 16384 type 182 action 2 ref_mod 1: -2 6.11.2) David Disseldorp
4 siblings, 2 replies; 43+ messages in thread
From: Marc MERLIN @ 2026-04-13 17:52 UTC (permalink / raw)
To: linux-btrfs, Boris Burkov, Josef Bacik, QuWenruo, Qu Wenruo,
Filipe Manana
Cc: Chris Murphy, Zygo Blaxell, Roman Mamedov, Su Yue, Su Yue
TL;DR: do I need to urgently disable simple quotas on all my fileystems
until I can upgrade to a confirmed fixed kernel?
Oh no, now it's my 2nd system with a btrfs crash, just days after I
enabled block-group-tree and simple quotas. This one is a simple
laptop without raid, and its backup filesystem crashed overnight likely
during balance (btrfs send/receive and snapshots, same than first one
below) crashed overnight.
First kernel was 6.12 which I can't upgrade, it's an rPi5 with vendor
kernels with special hardware support that's out of tree I think.
This one is kernel is 6.17.11 whic I will upgrade to 6.19.11+deb14 now
but I have no idea if it fixes anything.
I can't read the code or ooops outside of noticing the same exact
do_free_extent_accounting which can't be a coincidence.
I was able to rescue it with
merlin:~# mount -o rw,enospc_debug,skip_balance LABEL=btrfs_pool3 /mnt/btrfs_pool3
merlin:~# btrfs quota disable /mnt/btrfs_pool3
merlin:~# umount /mnt/btrfs_pool3
merlin:~# mount /mnt/btrfs_pool3
I'll reboot with the new kernel now, but I'm also now scared of simple quotas
since it's 2 crashes in 3 days, one seems not posislbe to recover from and a multi
week restore.
Suggestions welcome.
Possible analysis (could be wrong):
* __btrfs_free_extent & btrfs_qgroup_cleanup_dropped_subvolume:
Your btrfs-cleaner thread was in the middle of deleting an old snapshot. As it
was freeing the blocks (extents), it attempted to update the quota accounting
for that subvolume.
* failed to cleanup qgroup 0/83288: -2: The kernel tried to find the quota
group record for the subvolume being deleted, but the record was missing
or corrupted. To protect the filesystem, Btrfs panicked and locked
itself read-only.
------------[ cut here ]------------
BTRFS: Transaction aborted (error -2)
WARNING: CPU: 7 PID: 2987 at fs/btrfs/extent-tree.c:2999 __btrfs_free_extent.isra.0+0xc41/0x1020 [btrfs
Modules linked in: ftdi_sio usbserial ufs qnx4 hfsplus hfs cdrom minix msdos jfs nls_ucs2_utils xfs rpc
v4 dns_resolver nfs netfs rfcomm xt_tcpudp snd_seq_dummy snd_hrtimer xt_conntrack nf_conntrack_netlink
lgo xt_addrtype br_netfilter bridge stp llc ccm qrtr overlay cmac algif_hash algif_skcipher af_alg bnep
JECT nf_reject_ipv4 xt_MASQUERADE xt_LOG nf_log_syslog nft_compat nft_chain_nat nf_nat nf_conntrack nf_
efrag_ipv4 nf_tables binfmt_misc nls_ascii nls_cp437 vfat fat snd_soc_sof_sdw snd_soc_sdw_utils vboxdrv
11_sdca snd_soc_rt715_sdca snd_soc_rt1316_sdw dell_pc regmap_sdw_mbq platform_profile snd_hda_codec_int
mic regmap_sdw kvm snd_hda_intel btusb snd_sof_pci_intel_tgl iwlmvm snd_sof_pci_intel_cnl btrtl irqbypa
_hda_generic uvcvideo btintel soundwire_intel btbcm soundwire_generic_allocation videobuf2_vmalloc snd_
w_bpt btmtk uvc videobuf2_memops mac80211 snd_sof_intel_hda_common
videobuf2_v4l2 bluetooth snd_soc_hdac_hda videodev snd_sof_intel_hda_mlink videobuf2_common intel_uncore_frequency snd_sof_intel_hda libarc4 snd_hda_codec_hdmi snd_sof_pci intel_uncore_frequency_common mc ecdh_generic mei_hdcp soundwire_cadence snd_sof_xtensa_dsp mei_pxp ext4 x86_pkg_temp_thermal crc8 intel_rapl_msr processor_thermal_device_pci intel_powerclamp hid_sensor_als dell_laptop iwlwifi soundwire_bus processor_thermal_device dell_wmi hid_sensor_trigger processor_thermal_wt_hint rapl hid_sensor_iio_common platform_temperature_control snd_soc_avs industrialio_triggered_buffer crc16 snd_sof_probes processor_thermal_rfim intel_cstate iTCO_wdt mbcache dell_smbios kfifo_buf squashfs jbd2 loop dcdbas intel_uncore cfg80211 dell_smm_hwmon dell_wmi_sysman dell_wmi_ddv dell_wmi_descriptor firmware_attributes_class pcspkr snd_sof processor_thermal_rapl industrialio intel_pmc_bxt wmi_bmof snd_soc_hda_codec ucsi_acpi mei_me spd5118 iTCO_vendor_support intel_rapl_common snd_hda_ext_core snd_sof_utils typec_ucsi
snd_intel_dspcfg watchdog mei processor_thermal_wt_req snd_intel_sdw_acpi rfkill intel_pmc_core typec processor_thermal_power_floor snd_soc_skl_hda_dsp igen6_edac processor_thermal_mbox roles int3403_thermal pmt_telemetry snd_soc_intel_sof_board_helpers int340x_thermal_zone snd_soc_acpi_intel_match pmt_discovery joydev snd_soc_acpi_intel_sdca_quirks pmt_class snd_soc_acpi int3400_thermal intel_hid intel_pmc_ssram_telemetry snd_soc_sdca acpi_thermal_rel sparse_keymap acpi_tad acpi_pad ac serio_raw snd_soc_core evdev snd_compress snd_pcm_dmaengine snd_soc_intel_hda_dsp_common snd_hda_codec snd_hda_core snd_hwdep snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_midi snd_seq_midi_event snd_seq snd_timer snd_rawmidi snd_seq_device snd_ctl_led snd soundcore ac97_bus coretemp nfsd msr ecryptfs auth_rpcgss nvme_fabrics nfs_acl lockd efi_pstore grace sunrpc nfnetlink ip_tables x_tables autofs4 crc32c_cryptoapi essiv authenc btrfs blake2b_generic dm_crypt dm_mod efivarfs raid10 raid456 async_raid6_recov async_memcpy
async_pq async_xor async_tx xor raid6_pq raid1 raid0 md_mod sata_sil24 libata scsi_mod scsi_common e1000e r8169 realtek mdio_devres libphy mdio_bus mii xe configfs drm_gpusvm_helper drm_suballoc_helper hid_sensor_custom hid_sensor_hub intel_ishtp_hid nouveau mxm_wmi drm_gpuvm i915 gpu_sched drm_buddy drm_ttm_helper hid_multitouch ttm drm_exec i2c_algo_bit hid_generic drm_display_helper xhci_pci cec xhci_hcd rc_core nvme rtsx_pci_sdmmc i2c_hid_acpi intel_lpss_pci drm_client_lib i2c_hid nvme_core mmc_core intel_lpss usbcore video nvme_keyring i2c_i801 intel_ish_ipc drm_kms_helper hid ghash_clmulni_intel psmouse rtsx_pci thunderbolt intel_ishtp intel_vsec idma64 button usb_common i2c_smbus nvme_auth battery drm wmi aesni_intel
CPU: 7 UID: 0 PID: 2987 Comm: btrfs-cleaner Tainted: G U OE 6.17.11+deb14-amd64 #1 PREEMPT(lazy) Debian 6.17.11-1
Tainted: [U]=USER, [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
Hardware name: Dell Inc. XPS 17 9730/0JP3YK, BIOS 1.23.0 09/04/2025
RIP: 0010:__btrfs_free_extent.isra.0+0xc41/0x1020 [btrfs]
Code: ff ff 48 c7 c7 d8 ac 98 c1 e8 ab 7a 6c e4 0f 0b c6 44 24 2f 01 e9 22 8a 0e 00 8b 74 24 10 48 c7 c7 d8 ac 98 c1 e8 8f 7a 6c e4 <0f> 0b e9 50 ff ff ff 48 8b 34 24 48 8b 76 60 48 89 74 24 08 48 8d
RSP: 0000:ffffd372a10cfb40 EFLAGS: 00010246
RAX: 0000000000000000 RBX: 00000124f8b48000 RCX: 0000000000000027
RDX: ffff8d4d0f3dce48 RSI: 0000000000000001 RDI: ffff8d4d0f3dce40
RBP: 0000000000004000 R08: 0000000000000000 R09: ffffd372a10cf9e0
R10: ffff8d4d4f745068 R11: 00000000ffffdfff R12: 0000000000000000
R13: 000000000000002b R14: 00000000000039c1 R15: ffff8d4b42451930
FS: 0000000000000000(0000) GS:ffff8d4d669c8000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000563efa2028b8 CR3: 0000000ce5c2c006 CR4: 0000000000f70ef0
PKRU: 55555554
Call Trace:
<TASK>
__btrfs_run_delayed_refs+0x2dc/0xf70 [btrfs]
? read_block_for_search+0x19e/0x400 [btrfs]
? set_extent_buffer_dirty+0x26/0x200 [btrfs]
btrfs_run_delayed_refs+0x39/0x140 [btrfs]
btrfs_commit_transaction+0x6d/0xdf0 [btrfs]
btrfs_qgroup_cleanup_dropped_subvolume+0x49/0xb0 [btrfs]
btrfs_drop_snapshot+0x78e/0xcc0 [btrfs]
? __pfx_cleaner_kthread+0x10/0x10 [btrfs]
btrfs_clean_one_deleted_snapshot+0xc2/0x130 [btrfs]
cleaner_kthread+0xdc/0x160 [btrfs]
? __pfx_cleaner_kthread+0x10/0x10 [btrfs]
kthread+0xf9/0x240
? __pfx_kthread+0x10/0x10
ret_from_fork+0x194/0x1c0
? __pfx_kthread+0x10/0x10
ret_from_fork_asm+0x1a/0x30
</TASK>
---[ end trace 0000000000000000 ]---
BTRFS: error (device dm-4 state A) in do_free_extent_accounting:2999: errno=-2 No such entry
BTRFS info (device dm-4 state EA): forced readonly
BTRFS error (device dm-4 state EA): failed to run delayed ref for logical 1258303029248 num_bytes 16384 type 176 action 2 ref_mod 1: -2
BTRFS: error (device dm-4 state EA) in btrfs_run_delayed_refs:2161: errno=-2 No such entry
BTRFS warning (device dm-4 state EA): failed to cleanup qgroup 0/83288: -2
On Fri, Apr 10, 2026 at 08:35:33PM -0700, Marc MERLIN wrote:
>
> It started with:
> [23345.326321] BTRFS: error (device dm-0 state A) in do_free_extent_accounting:2996: errno=-2 No such entry
> [23345.336394] BTRFS error (device dm-0 state EA): failed to run delayed ref for logical 15506102321152 num_bytes 16384 type 182 action 2 ref_mod 1: -2
> [23345.350299] BTRFS: error (device dm-0 state EA) in btrfs_run_delayed_refs:2215: errno=-2 No such entry
> [23345.360154] BTRFS warning (device dm-0 state EA):
>
> I ended up with:
>
> moremagic:~# mount -t btrfs -o rw,skip_balance,space_cache=v2,clear_cache /dev/mapper/crypt_bcache0 /mnt/btrfs_bigbackup
> BTRFS: device label DS6 devid 1 transid 296950 /dev/mapper/crypt_bcache0 (251:0) scanned by mount (6029)
> BTRFS info (device dm-0): first mount of filesystem a97dec85-a0d5-42ab-a0ef-e9b7479fbe43
> BTRFS info (device dm-0): using crc32c (crc32c-generic) checksum algorithm
> BTRFS warning (device dm-0): read-write for sector size 4096 with page size 16384 is experimental
> BTRFS info (device dm-0): bdev /dev/mapper/crypt_bcache0 errs: wr 0, rd 0, flush 0, corrupt 5074, gen 0
> ------------[ cut here ]------------
> BTRFS: Transaction aborted (error -2)
> WARNING: CPU: 3 PID: 6029 at fs/btrfs/extent-tree.c:2996 __btrfs_free_extent.isra.0+0x13a0/0x14a0 [btrfs]
> Modules linked in: dm_crypt dm_mod bcache raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xt_MASQUERADE ipt_REJECT nf_reject_ipv4 xt_tcpudp xt_conntrack xt_LOG nf_log_syslog nft_compat nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nf_tables rfcomm algif_hash algif_skcipher af_alg bnep cp210x brcmfmac_wcc binfmt_misc usbserial hci_uart brcmfmac btbcm vc4 snd_soc_hdmi_codec brcmutil bluetooth drm_display_helper cfg80211 cec drm_dma_helper rpi_hevc_dec ecdh_generic v4l2_mem2mem ecc snd_soc_core pisp_be videobuf2_dma_contig v3d videobuf2_memops videobuf2_v4l2 gpu_sched rfkill videodev drm_shmem_helper snd_compress snd_pcm_dmaengine snd_pcm videobuf2_common rp1_pio snd_timer snd drm_kms_helper mc raspberrypi_gpiomem rp1_fw sg sch_fq_codel ecryptfs fuse drm drm_panel_orientation_quirks backlight nfnetlink ip_tables x_tables raid1 aes_ce_blk aes_ce_cipher ghash_ce gf128mul libaes sha2_ce spidev sha256_arm64 sha1_ce raspberrypi_hwmon sha1_generic ahci i2c_brcmstb spi_bcm2835
> md_mod gpio_keys libahci pwm_fan rp1_adc libata rp1_mailbox nvmem_rmem uio_pdrv_genirq uio btrfs blake2b_generic xor xor_neon raid6_pq zram lz4_compress ipv6
> CPU: 3 UID: 0 PID: 6029 Comm: mount Not tainted 6.12.47+rpt-rpi-2712 #1 Debian 1:6.12.47-1+rpt1
> Hardware name: Raspberry Pi 5 Model B Rev 1.1 (DT)
> pstate: 60400009 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
> pc : __btrfs_free_extent.isra.0+0x13a0/0x14a0 [btrfs]
> lr : __btrfs_free_extent.isra.0+0x13a0/0x14a0 [btrfs]
> sp : ffffc000868bb680
> x29: ffffc000868bb720 x28: 0000000000000000 x27: 0000000000002f02
> x26: 000000000000007f x25: ffff8001de833aa0 x24: 0000000000004000
> x23: 0000000000000000 x22: ffff800102b64e70 x21: 0000000000004000
> x20: 00000e1a4bb88000 x19: 00000000fffffffe x18: 0000000000000000
> x17: 0000000000000000 x16: 0000000000000000 x15: 0000000000000000
> x14: 0000000000000000 x13: 0000000000000000 x12: 0000000000000000
> x11: 00000000000000c0 x10: 0000000000001a40 x9 : ffffd06fce4e06c0
> x8 : ffff80011f56e0a0 x7 : 000000042f72a7bd x6 : 0000000000000039
> x5 : 0000000000000001 x4 : 0000000000001ab0 x3 : 0000000000000804
> x2 : 0000000000000000 x1 : 0000000000000000 x0 : ffff80011f56c600
> Call trace:
> __btrfs_free_extent.isra.0+0x13a0/0x14a0 [btrfs]
> __btrfs_run_delayed_refs+0x508/0xec0 [btrfs]
> btrfs_run_delayed_refs+0x48/0x198 [btrfs]
> btrfs_commit_transaction+0x88/0xe20 [btrfs]
> btrfs_recover_relocation+0x55c/0x5d0 [btrfs]
> btrfs_start_pre_rw_mount+0x1d4/0x470 [btrfs]
> open_ctree+0x101c/0x13b8 [btrfs]
> btrfs_get_tree+0x5b4/0x800 [btrfs]
> vfs_get_tree+0x30/0x108
> fc_mount+0x20/0x68
> btrfs_get_tree+0x238/0x800 [btrfs]
> vfs_get_tree+0x30/0x108
> vfs_cmd_create+0x58/0xf8
> __arm64_sys_fsconfig+0x444/0x5b8
> invoke_syscall+0x50/0x120
> el0_svc_common.constprop.0+0x48/0xf0
> do_el0_svc+0x24/0x38
> el0_svc+0x30/0xf8
> el0t_64_sync_handler+0x120/0x130
> el0t_64_sync+0x190/0x198
> ---[ end trace 0000000000000000 ]---
> BTRFS: error (device dm-0 state A) in do_free_extent_accounting:2996: errno=-2 No such entry
> BTRFS error (device dm-0 state EA): failed to run delayed ref for logical 15506102321152 num_bytes 16384 type 182 action 2 ref_mod 1: -2
> BTRFS: error (device dm-0 state EA) in btrfs_run_delayed_refs:2215: errno=-2 No such entry
> BTRFS warning (device dm-0 state EA): failed to recover relocation: -2
> BTRFS error (device dm-0 state EA): commit super ret -30
> BTRFS error (device dm-0 state EA): open_ctree failed: -2
>
>
> Gemini said
>
> The Btrfs "Ghost" Accounting When you added discard=async to your fstab
> (or remounted with it), you told the Btrfs kernel module to start a specific
> background thread.
> Btrfs's Perspective: "The user told me to use async discard. I will now start a
> list of every extent we delete so I can 'trim' them later in the background."
> The Problem: Btrfs doesn't check if the underlying dm-crypt device actually
> supports discards before it starts its own internal accounting.
> The Result: Btrfs started tracking a massive list of "extents to be discarded"
> in its memory and metadata.
>
> 2. The "No Such Entry" (-2) Race Condition The crash didn't happen because a
> command hit a drive; it happened because of a logic race inside the kernel's
> Btrfs code:
> The Balance Thread: You were running a balance. This thread moves data from "Old
> Block A" to "New Block B."
> The Discard Thread: Because discard=async was on, the discard thread saw "Old
> Block A" get freed. It put "Old Block A" on its "to-do list."
> The Metadata Conflict: The balance thread finished moving the data and
> successfully deleted the reference to "Old Block A" from the extent tree.
> The Crash: A few milliseconds later, the async discard thread woke up and tried
> to "pin" or "process" the metadata for "Old Block A." It looked in the tree,
> found nothing (because the balance already deleted it), and threw an ENOENT
> (Error -2: No such entry).
> Btrfs panicked: "Wait, I was told to discard this block, but it doesn't exist in
> my records anymore! Something is inconsistent!" → Transaction Abort.
>
> more details:
> backuproot didn't work (read write)
> I was forced to run
> btrfstune --convert-from-block-group-tree /dev/mapper/crypt_bcache0
> because
> When you ran btrfs check --clear-space-cache v2, the tool did exactly
> what it was supposed to do: it deleted the Free Space Tree and removed
> the FREE_SPACE_TREE flag from your superblock.
> The Conflict: Your 23TB array was formatted with the modern
> block-group-tree feature (which speeds up mounting).
> The Kernel Rule: The Btrfs kernel code explicitly dictates: If the Block
> Group Tree is enabled, the Free Space Tree MUST also be enabled. * The
> Crash: Because the FREE_SPACE_TREE flag is now missing, the kernel sees
> an "illegal" superblock state and throws a fatal -22 error, refusing to
> proceed to the mount options.
>
> This was vexing, hours lost removing the block group tree.
> and when it was finally finished,
> mount -t btrfs -o skip_balance /dev/mapper/crypt_bcache0 /mnt/btrfs_bigbackup/
> did run, but crashed as above
>
> Now doing a repair in case it can salvage things.
>
> Marc
> --
> "A mouse is a device used to point at the xterm you want to type in" - A.S.R.
>
> Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
>
--
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
^ permalink raw reply [flat|nested] 43+ messages in thread* Simple quota unsafe? RIP: 0010:__btrfs_free_extent.isra.0+0xc41/0x1020 [btrfs] / do_free_extent_accounting:2999: errno=-2 No such entry
2026-04-13 17:52 ` Simple quota unsafe? RIP: 0010:__btrfs_free_extent.isra.0+0xc41/0x1020 [btrfs] / do_free_extent_accounting:2999: errno=-2 No such entry Marc MERLIN
@ 2026-04-13 17:52 ` Marc MERLIN
2026-04-13 18:47 ` Boris Burkov
1 sibling, 0 replies; 43+ messages in thread
From: Marc MERLIN @ 2026-04-13 17:52 UTC (permalink / raw)
To: linux-btrfs, Boris Burkov, Josef Bacik, QuWenruo, Qu Wenruo,
Filipe Manana
Cc: Chris Murphy, Zygo Blaxell, Roman Mamedov, Su Yue, Su Yue
TL;DR: do I need to urgently disable simple quotas on all my fileystems
until I can upgrade to a confirmed fixed kernel?
Oh no, now it's my 2nd system with a btrfs crash, just days after I
enabled block-group-tree and simple quotas. This one is a simple
laptop without raid, and its backup filesystem crashed overnight likely
during balance (btrfs send/receive and snapshots, same than first one
below) crashed overnight.
First kernel was 6.12 which I can't upgrade, it's an rPi5 with vendor
kernels with special hardware support that's out of tree I think.
This one is kernel is 6.17.11 whic I will upgrade to 6.19.11+deb14 now
but I have no idea if it fixes anything.
I can't read the code or ooops outside of noticing the same exact
do_free_extent_accounting which can't be a coincidence.
I was able to rescue it with
merlin:~# mount -o rw,enospc_debug,skip_balance LABEL=btrfs_pool3 /mnt/btrfs_pool3
merlin:~# btrfs quota disable /mnt/btrfs_pool3
merlin:~# umount /mnt/btrfs_pool3
merlin:~# mount /mnt/btrfs_pool3
I'll reboot with the new kernel now, but I'm also now scared of simple quotas
since it's 2 crashes in 3 days, one seems not posislbe to recover from and a multi
week restore.
Suggestions welcome.
Possible analysis (could be wrong):
* __btrfs_free_extent & btrfs_qgroup_cleanup_dropped_subvolume:
Your btrfs-cleaner thread was in the middle of deleting an old snapshot. As it
was freeing the blocks (extents), it attempted to update the quota accounting
for that subvolume.
* failed to cleanup qgroup 0/83288: -2: The kernel tried to find the quota
group record for the subvolume being deleted, but the record was missing
or corrupted. To protect the filesystem, Btrfs panicked and locked
itself read-only.
------------[ cut here ]------------
BTRFS: Transaction aborted (error -2)
WARNING: CPU: 7 PID: 2987 at fs/btrfs/extent-tree.c:2999 __btrfs_free_extent.isra.0+0xc41/0x1020 [btrfs
Modules linked in: ftdi_sio usbserial ufs qnx4 hfsplus hfs cdrom minix msdos jfs nls_ucs2_utils xfs rpc
v4 dns_resolver nfs netfs rfcomm xt_tcpudp snd_seq_dummy snd_hrtimer xt_conntrack nf_conntrack_netlink
lgo xt_addrtype br_netfilter bridge stp llc ccm qrtr overlay cmac algif_hash algif_skcipher af_alg bnep
JECT nf_reject_ipv4 xt_MASQUERADE xt_LOG nf_log_syslog nft_compat nft_chain_nat nf_nat nf_conntrack nf_
efrag_ipv4 nf_tables binfmt_misc nls_ascii nls_cp437 vfat fat snd_soc_sof_sdw snd_soc_sdw_utils vboxdrv
11_sdca snd_soc_rt715_sdca snd_soc_rt1316_sdw dell_pc regmap_sdw_mbq platform_profile snd_hda_codec_int
mic regmap_sdw kvm snd_hda_intel btusb snd_sof_pci_intel_tgl iwlmvm snd_sof_pci_intel_cnl btrtl irqbypa
_hda_generic uvcvideo btintel soundwire_intel btbcm soundwire_generic_allocation videobuf2_vmalloc snd_
w_bpt btmtk uvc videobuf2_memops mac80211 snd_sof_intel_hda_common
videobuf2_v4l2 bluetooth snd_soc_hdac_hda videodev snd_sof_intel_hda_mlink videobuf2_common intel_uncore_frequency snd_sof_intel_hda libarc4 snd_hda_codec_hdmi snd_sof_pci intel_uncore_frequency_common mc ecdh_generic mei_hdcp soundwire_cadence snd_sof_xtensa_dsp mei_pxp ext4 x86_pkg_temp_thermal crc8 intel_rapl_msr processor_thermal_device_pci intel_powerclamp hid_sensor_als dell_laptop iwlwifi soundwire_bus processor_thermal_device dell_wmi hid_sensor_trigger processor_thermal_wt_hint rapl hid_sensor_iio_common platform_temperature_control snd_soc_avs industrialio_triggered_buffer crc16 snd_sof_probes processor_thermal_rfim intel_cstate iTCO_wdt mbcache dell_smbios kfifo_buf squashfs jbd2 loop dcdbas intel_uncore cfg80211 dell_smm_hwmon dell_wmi_sysman dell_wmi_ddv dell_wmi_descriptor firmware_attributes_class pcspkr snd_sof processor_thermal_rapl industrialio intel_pmc_bxt wmi_bmof snd_soc_hda_codec ucsi_acpi mei_me spd5118 iTCO_vendor_support intel_rapl_common snd_hda_ext_core snd_sof_utils typec_ucsi
snd_intel_dspcfg watchdog mei processor_thermal_wt_req snd_intel_sdw_acpi rfkill intel_pmc_core typec processor_thermal_power_floor snd_soc_skl_hda_dsp igen6_edac processor_thermal_mbox roles int3403_thermal pmt_telemetry snd_soc_intel_sof_board_helpers int340x_thermal_zone snd_soc_acpi_intel_match pmt_discovery joydev snd_soc_acpi_intel_sdca_quirks pmt_class snd_soc_acpi int3400_thermal intel_hid intel_pmc_ssram_telemetry snd_soc_sdca acpi_thermal_rel sparse_keymap acpi_tad acpi_pad ac serio_raw snd_soc_core evdev snd_compress snd_pcm_dmaengine snd_soc_intel_hda_dsp_common snd_hda_codec snd_hda_core snd_hwdep snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_midi snd_seq_midi_event snd_seq snd_timer snd_rawmidi snd_seq_device snd_ctl_led snd soundcore ac97_bus coretemp nfsd msr ecryptfs auth_rpcgss nvme_fabrics nfs_acl lockd efi_pstore grace sunrpc nfnetlink ip_tables x_tables autofs4 crc32c_cryptoapi essiv authenc btrfs blake2b_generic dm_crypt dm_mod efivarfs raid10 raid456 async_raid6_recov async_memcpy
async_pq async_xor async_tx xor raid6_pq raid1 raid0 md_mod sata_sil24 libata scsi_mod scsi_common e1000e r8169 realtek mdio_devres libphy mdio_bus mii xe configfs drm_gpusvm_helper drm_suballoc_helper hid_sensor_custom hid_sensor_hub intel_ishtp_hid nouveau mxm_wmi drm_gpuvm i915 gpu_sched drm_buddy drm_ttm_helper hid_multitouch ttm drm_exec i2c_algo_bit hid_generic drm_display_helper xhci_pci cec xhci_hcd rc_core nvme rtsx_pci_sdmmc i2c_hid_acpi intel_lpss_pci drm_client_lib i2c_hid nvme_core mmc_core intel_lpss usbcore video nvme_keyring i2c_i801 intel_ish_ipc drm_kms_helper hid ghash_clmulni_intel psmouse rtsx_pci thunderbolt intel_ishtp intel_vsec idma64 button usb_common i2c_smbus nvme_auth battery drm wmi aesni_intel
CPU: 7 UID: 0 PID: 2987 Comm: btrfs-cleaner Tainted: G U OE 6.17.11+deb14-amd64 #1 PREEMPT(lazy) Debian 6.17.11-1
Tainted: [U]=USER, [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
Hardware name: Dell Inc. XPS 17 9730/0JP3YK, BIOS 1.23.0 09/04/2025
RIP: 0010:__btrfs_free_extent.isra.0+0xc41/0x1020 [btrfs]
Code: ff ff 48 c7 c7 d8 ac 98 c1 e8 ab 7a 6c e4 0f 0b c6 44 24 2f 01 e9 22 8a 0e 00 8b 74 24 10 48 c7 c7 d8 ac 98 c1 e8 8f 7a 6c e4 <0f> 0b e9 50 ff ff ff 48 8b 34 24 48 8b 76 60 48 89 74 24 08 48 8d
RSP: 0000:ffffd372a10cfb40 EFLAGS: 00010246
RAX: 0000000000000000 RBX: 00000124f8b48000 RCX: 0000000000000027
RDX: ffff8d4d0f3dce48 RSI: 0000000000000001 RDI: ffff8d4d0f3dce40
RBP: 0000000000004000 R08: 0000000000000000 R09: ffffd372a10cf9e0
R10: ffff8d4d4f745068 R11: 00000000ffffdfff R12: 0000000000000000
R13: 000000000000002b R14: 00000000000039c1 R15: ffff8d4b42451930
FS: 0000000000000000(0000) GS:ffff8d4d669c8000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000563efa2028b8 CR3: 0000000ce5c2c006 CR4: 0000000000f70ef0
PKRU: 55555554
Call Trace:
<TASK>
__btrfs_run_delayed_refs+0x2dc/0xf70 [btrfs]
? read_block_for_search+0x19e/0x400 [btrfs]
? set_extent_buffer_dirty+0x26/0x200 [btrfs]
btrfs_run_delayed_refs+0x39/0x140 [btrfs]
btrfs_commit_transaction+0x6d/0xdf0 [btrfs]
btrfs_qgroup_cleanup_dropped_subvolume+0x49/0xb0 [btrfs]
btrfs_drop_snapshot+0x78e/0xcc0 [btrfs]
? __pfx_cleaner_kthread+0x10/0x10 [btrfs]
btrfs_clean_one_deleted_snapshot+0xc2/0x130 [btrfs]
cleaner_kthread+0xdc/0x160 [btrfs]
? __pfx_cleaner_kthread+0x10/0x10 [btrfs]
kthread+0xf9/0x240
? __pfx_kthread+0x10/0x10
ret_from_fork+0x194/0x1c0
? __pfx_kthread+0x10/0x10
ret_from_fork_asm+0x1a/0x30
</TASK>
---[ end trace 0000000000000000 ]---
BTRFS: error (device dm-4 state A) in do_free_extent_accounting:2999: errno=-2 No such entry
BTRFS info (device dm-4 state EA): forced readonly
BTRFS error (device dm-4 state EA): failed to run delayed ref for logical 1258303029248 num_bytes 16384 type 176 action 2 ref_mod 1: -2
BTRFS: error (device dm-4 state EA) in btrfs_run_delayed_refs:2161: errno=-2 No such entry
BTRFS warning (device dm-4 state EA): failed to cleanup qgroup 0/83288: -2
On Fri, Apr 10, 2026 at 08:35:33PM -0700, Marc MERLIN wrote:
>
> It started with:
> [23345.326321] BTRFS: error (device dm-0 state A) in do_free_extent_accounting:2996: errno=-2 No such entry
> [23345.336394] BTRFS error (device dm-0 state EA): failed to run delayed ref for logical 15506102321152 num_bytes 16384 type 182 action 2 ref_mod 1: -2
> [23345.350299] BTRFS: error (device dm-0 state EA) in btrfs_run_delayed_refs:2215: errno=-2 No such entry
> [23345.360154] BTRFS warning (device dm-0 state EA):
>
> I ended up with:
>
> moremagic:~# mount -t btrfs -o rw,skip_balance,space_cache=v2,clear_cache /dev/mapper/crypt_bcache0 /mnt/btrfs_bigbackup
> BTRFS: device label DS6 devid 1 transid 296950 /dev/mapper/crypt_bcache0 (251:0) scanned by mount (6029)
> BTRFS info (device dm-0): first mount of filesystem a97dec85-a0d5-42ab-a0ef-e9b7479fbe43
> BTRFS info (device dm-0): using crc32c (crc32c-generic) checksum algorithm
> BTRFS warning (device dm-0): read-write for sector size 4096 with page size 16384 is experimental
> BTRFS info (device dm-0): bdev /dev/mapper/crypt_bcache0 errs: wr 0, rd 0, flush 0, corrupt 5074, gen 0
> ------------[ cut here ]------------
> BTRFS: Transaction aborted (error -2)
> WARNING: CPU: 3 PID: 6029 at fs/btrfs/extent-tree.c:2996 __btrfs_free_extent.isra.0+0x13a0/0x14a0 [btrfs]
> Modules linked in: dm_crypt dm_mod bcache raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xt_MASQUERADE ipt_REJECT nf_reject_ipv4 xt_tcpudp xt_conntrack xt_LOG nf_log_syslog nft_compat nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nf_tables rfcomm algif_hash algif_skcipher af_alg bnep cp210x brcmfmac_wcc binfmt_misc usbserial hci_uart brcmfmac btbcm vc4 snd_soc_hdmi_codec brcmutil bluetooth drm_display_helper cfg80211 cec drm_dma_helper rpi_hevc_dec ecdh_generic v4l2_mem2mem ecc snd_soc_core pisp_be videobuf2_dma_contig v3d videobuf2_memops videobuf2_v4l2 gpu_sched rfkill videodev drm_shmem_helper snd_compress snd_pcm_dmaengine snd_pcm videobuf2_common rp1_pio snd_timer snd drm_kms_helper mc raspberrypi_gpiomem rp1_fw sg sch_fq_codel ecryptfs fuse drm drm_panel_orientation_quirks backlight nfnetlink ip_tables x_tables raid1 aes_ce_blk aes_ce_cipher ghash_ce gf128mul libaes sha2_ce spidev sha256_arm64 sha1_ce raspberrypi_hwmon sha1_generic ahci i2c_brcmstb spi_bcm2835
> md_mod gpio_keys libahci pwm_fan rp1_adc libata rp1_mailbox nvmem_rmem uio_pdrv_genirq uio btrfs blake2b_generic xor xor_neon raid6_pq zram lz4_compress ipv6
> CPU: 3 UID: 0 PID: 6029 Comm: mount Not tainted 6.12.47+rpt-rpi-2712 #1 Debian 1:6.12.47-1+rpt1
> Hardware name: Raspberry Pi 5 Model B Rev 1.1 (DT)
> pstate: 60400009 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
> pc : __btrfs_free_extent.isra.0+0x13a0/0x14a0 [btrfs]
> lr : __btrfs_free_extent.isra.0+0x13a0/0x14a0 [btrfs]
> sp : ffffc000868bb680
> x29: ffffc000868bb720 x28: 0000000000000000 x27: 0000000000002f02
> x26: 000000000000007f x25: ffff8001de833aa0 x24: 0000000000004000
> x23: 0000000000000000 x22: ffff800102b64e70 x21: 0000000000004000
> x20: 00000e1a4bb88000 x19: 00000000fffffffe x18: 0000000000000000
> x17: 0000000000000000 x16: 0000000000000000 x15: 0000000000000000
> x14: 0000000000000000 x13: 0000000000000000 x12: 0000000000000000
> x11: 00000000000000c0 x10: 0000000000001a40 x9 : ffffd06fce4e06c0
> x8 : ffff80011f56e0a0 x7 : 000000042f72a7bd x6 : 0000000000000039
> x5 : 0000000000000001 x4 : 0000000000001ab0 x3 : 0000000000000804
> x2 : 0000000000000000 x1 : 0000000000000000 x0 : ffff80011f56c600
> Call trace:
> __btrfs_free_extent.isra.0+0x13a0/0x14a0 [btrfs]
> __btrfs_run_delayed_refs+0x508/0xec0 [btrfs]
> btrfs_run_delayed_refs+0x48/0x198 [btrfs]
> btrfs_commit_transaction+0x88/0xe20 [btrfs]
> btrfs_recover_relocation+0x55c/0x5d0 [btrfs]
> btrfs_start_pre_rw_mount+0x1d4/0x470 [btrfs]
> open_ctree+0x101c/0x13b8 [btrfs]
> btrfs_get_tree+0x5b4/0x800 [btrfs]
> vfs_get_tree+0x30/0x108
> fc_mount+0x20/0x68
> btrfs_get_tree+0x238/0x800 [btrfs]
> vfs_get_tree+0x30/0x108
> vfs_cmd_create+0x58/0xf8
> __arm64_sys_fsconfig+0x444/0x5b8
> invoke_syscall+0x50/0x120
> el0_svc_common.constprop.0+0x48/0xf0
> do_el0_svc+0x24/0x38
> el0_svc+0x30/0xf8
> el0t_64_sync_handler+0x120/0x130
> el0t_64_sync+0x190/0x198
> ---[ end trace 0000000000000000 ]---
> BTRFS: error (device dm-0 state A) in do_free_extent_accounting:2996: errno=-2 No such entry
> BTRFS error (device dm-0 state EA): failed to run delayed ref for logical 15506102321152 num_bytes 16384 type 182 action 2 ref_mod 1: -2
> BTRFS: error (device dm-0 state EA) in btrfs_run_delayed_refs:2215: errno=-2 No such entry
> BTRFS warning (device dm-0 state EA): failed to recover relocation: -2
> BTRFS error (device dm-0 state EA): commit super ret -30
> BTRFS error (device dm-0 state EA): open_ctree failed: -2
>
>
> Gemini said
>
> The Btrfs "Ghost" Accounting When you added discard=async to your fstab
> (or remounted with it), you told the Btrfs kernel module to start a specific
> background thread.
> Btrfs's Perspective: "The user told me to use async discard. I will now start a
> list of every extent we delete so I can 'trim' them later in the background."
> The Problem: Btrfs doesn't check if the underlying dm-crypt device actually
> supports discards before it starts its own internal accounting.
> The Result: Btrfs started tracking a massive list of "extents to be discarded"
> in its memory and metadata.
>
> 2. The "No Such Entry" (-2) Race Condition The crash didn't happen because a
> command hit a drive; it happened because of a logic race inside the kernel's
> Btrfs code:
> The Balance Thread: You were running a balance. This thread moves data from "Old
> Block A" to "New Block B."
> The Discard Thread: Because discard=async was on, the discard thread saw "Old
> Block A" get freed. It put "Old Block A" on its "to-do list."
> The Metadata Conflict: The balance thread finished moving the data and
> successfully deleted the reference to "Old Block A" from the extent tree.
> The Crash: A few milliseconds later, the async discard thread woke up and tried
> to "pin" or "process" the metadata for "Old Block A." It looked in the tree,
> found nothing (because the balance already deleted it), and threw an ENOENT
> (Error -2: No such entry).
> Btrfs panicked: "Wait, I was told to discard this block, but it doesn't exist in
> my records anymore! Something is inconsistent!" → Transaction Abort.
>
> more details:
> backuproot didn't work (read write)
> I was forced to run
> btrfstune --convert-from-block-group-tree /dev/mapper/crypt_bcache0
> because
> When you ran btrfs check --clear-space-cache v2, the tool did exactly
> what it was supposed to do: it deleted the Free Space Tree and removed
> the FREE_SPACE_TREE flag from your superblock.
> The Conflict: Your 23TB array was formatted with the modern
> block-group-tree feature (which speeds up mounting).
> The Kernel Rule: The Btrfs kernel code explicitly dictates: If the Block
> Group Tree is enabled, the Free Space Tree MUST also be enabled. * The
> Crash: Because the FREE_SPACE_TREE flag is now missing, the kernel sees
> an "illegal" superblock state and throws a fatal -22 error, refusing to
> proceed to the mount options.
>
> This was vexing, hours lost removing the block group tree.
> and when it was finally finished,
> mount -t btrfs -o skip_balance /dev/mapper/crypt_bcache0 /mnt/btrfs_bigbackup/
> did run, but crashed as above
>
> Now doing a repair in case it can salvage things.
>
> Marc
> --
> "A mouse is a device used to point at the xterm you want to type in" - A.S.R.
>
> Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
>
--
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: Simple quota unsafe? RIP: 0010:__btrfs_free_extent.isra.0+0xc41/0x1020 [btrfs] / do_free_extent_accounting:2999: errno=-2 No such entry
2026-04-13 17:52 ` Simple quota unsafe? RIP: 0010:__btrfs_free_extent.isra.0+0xc41/0x1020 [btrfs] / do_free_extent_accounting:2999: errno=-2 No such entry Marc MERLIN
2026-04-13 17:52 ` Marc MERLIN
@ 2026-04-13 18:47 ` Boris Burkov
2026-04-13 19:40 ` Marc MERLIN
1 sibling, 1 reply; 43+ messages in thread
From: Boris Burkov @ 2026-04-13 18:47 UTC (permalink / raw)
To: Marc MERLIN
Cc: linux-btrfs, Josef Bacik, QuWenruo, Qu Wenruo, Filipe Manana,
Chris Murphy, Zygo Blaxell, Roman Mamedov, Su Yue, Su Yue
On Mon, Apr 13, 2026 at 10:52:53AM -0700, Marc MERLIN wrote:
> TL;DR: do I need to urgently disable simple quotas on all my fileystems
> until I can upgrade to a confirmed fixed kernel?
>
> Oh no, now it's my 2nd system with a btrfs crash, just days after I
> enabled block-group-tree and simple quotas. This one is a simple
> laptop without raid, and its backup filesystem crashed overnight likely
> during balance (btrfs send/receive and snapshots, same than first one
> below) crashed overnight.
>
> First kernel was 6.12 which I can't upgrade, it's an rPi5 with vendor
> kernels with special hardware support that's out of tree I think.
> This one is kernel is 6.17.11 whic I will upgrade to 6.19.11+deb14 now
> but I have no idea if it fixes anything.
>
> I can't read the code or ooops outside of noticing the same exact
> do_free_extent_accounting which can't be a coincidence.
>
> I was able to rescue it with
> merlin:~# mount -o rw,enospc_debug,skip_balance LABEL=btrfs_pool3 /mnt/btrfs_pool3
> merlin:~# btrfs quota disable /mnt/btrfs_pool3
> merlin:~# umount /mnt/btrfs_pool3
> merlin:~# mount /mnt/btrfs_pool3
>
> I'll reboot with the new kernel now, but I'm also now scared of simple quotas
> since it's 2 crashes in 3 days, one seems not posislbe to recover from and a multi
> week restore.
> Suggestions welcome.
I am currently a little confused about your full story, so please help
me make sure I understand. I would like to fix any squotas problems you
are seeing if possible. I'm going to restate what I have understood from
your reports to try to confirm I am following properly.
You sent this email in reply to your previous email which I understand
was for the following trace:
2026-04-10T15:40:14.846141-07:00 moremagic kernel: Call trace:
2026-04-10T15:40:14.846143-07:00 moremagic kernel: __btrfs_free_extent.isra.0+0x13a0/0x14a0 [btrfs]
2026-04-10T15:40:14.846145-07:00 moremagic kernel: __btrfs_run_delayed_refs+0x508/0xec0 [btrfs]
2026-04-10T15:40:14.846146-07:00 moremagic kernel: btrfs_run_delayed_refs+0x48/0x198 [btrfs]
2026-04-10T15:40:14.846148-07:00 moremagic kernel: btrfs_commit_transaction+0x88/0xe20 [btrfs]
2026-04-10T15:40:14.846149-07:00 moremagic kernel: relocate_block_group+0x174/0x508 [btrfs]
2026-04-10T15:40:14.846150-07:00 moremagic kernel: btrfs_relocate_block_group+0x228/0x3d8 [btrfs]
2026-04-10T15:40:14.846151-07:00 moremagic kernel: btrfs_relocate_chunk+0x44/0x158 [btrfs]
2026-04-10T15:40:14.846153-07:00 moremagic kernel: btrfs_balance+0x734/0x1000 [btrfs]
2026-04-10T15:40:14.846154-07:00 moremagic kernel: balance_kthread+0xbc/0x1f0 [btrfs]
2026-04-10T15:40:14.846156-07:00 moremagic kernel: kthread+0x118/0x128
2026-04-10T15:40:14.846157-07:00 moremagic kernel: ret_from_fork+0x10/0x20
2026-04-10T15:40:14.846159-07:00 moremagic kernel: ---[ end trace 0000000000000000 ]---
2026-04-10T15:40:14.846170-07:00 moremagic kernel: BTRFS: error (device dm-0 state A) in do_free_extent_accounting:2996: errno=-2 No such entry
2026-04-10T15:40:14.869757-07:00 moremagic kernel: BTRFS info (device dm-0 state EA): forced readonly
2026-04-10T15:40:14.870327-07:00 moremagic kernel: BTRFS error (device dm-0 state EA): failed to run delayed ref for logical 15506102321152 num_bytes 16384 type 182 action 2 ref_mod 1: -2
I will call this report 1. Report 1 is from a rpi running 6.12 with possible out
of tree modules and raid5.
And now you are seeing a different path abort on a very similar invalid
state, also in __btrfs_free_extent, but this time called via qgroups
logic triggering a transaction commit:
Call Trace:
<TASK>
__btrfs_run_delayed_refs+0x2dc/0xf70 [btrfs]
? read_block_for_search+0x19e/0x400 [btrfs]
? set_extent_buffer_dirty+0x26/0x200 [btrfs]
btrfs_run_delayed_refs+0x39/0x140 [btrfs]
btrfs_commit_transaction+0x6d/0xdf0 [btrfs]
btrfs_qgroup_cleanup_dropped_subvolume+0x49/0xb0 [btrfs]
btrfs_drop_snapshot+0x78e/0xcc0 [btrfs]
? __pfx_cleaner_kthread+0x10/0x10 [btrfs]
btrfs_clean_one_deleted_snapshot+0xc2/0x130 [btrfs]
cleaner_kthread+0xdc/0x160 [btrfs]
? __pfx_cleaner_kthread+0x10/0x10 [btrfs]
kthread+0xf9/0x240
? __pfx_kthread+0x10/0x10
ret_from_fork+0x194/0x1c0
? __pfx_kthread+0x10/0x10
ret_from_fork_asm+0x1a/0x30
</TASK>
---[ end trace 0000000000000000 ]---
BTRFS: error (device dm-4 state A) in do_free_extent_accounting:2999: errno=-2 No such entry
BTRFS info (device dm-4 state EA): forced readonly
BTRFS error (device dm-4 state EA): failed to run delayed ref for logical 1258303029248 num_bytes 16384 type 176 action 2 ref_mod 1: -2
I'll call this report 2. Report 2 is from a laptop with no fancy raid
and upstream kernel 6.17.
Is that all accurate?
Some further questions/observations:
- I noticed that your paste from report 1 (https://pastebin.com/7HmQwy3n)
had 16k pages and 4k block size:
2026-04-10T10:43:22.673638-07:00 moremagic kernel: BTRFS warning (device dm-0): read-write for sector size 4096 with page size 16384 is experimental
which seems a bit risky on an old kernel. There were a lot of fixes for
subpage block size support in recent kernels. I believe it has been
quite stable for us on 6.16 but Qu can give the most authoritative
answer on when that got solid.
- Is the laptop also running subpage block size? Do you have a full
dmesg from that system which you can share?
- On which of these systems did you enable squotas and when?
- The squotas specific object is type 172, and neither of these aborts
reference that (182 is BTRFS_SHARED_BLOCK_REF_KEY and 176 is
BTRFS_TREE_BLOCK_REF_KEY). So that noticeably reduces my suspicion of
squotas at this point. As far as I can tell, the main things connecting
us to squotas are: the bug is hit in __btrfs_free_extent and the bug was
hit once in qgroup code while deleting a snapshot. The former is still
kind of interesting, especially if you enabled squotas very recently
on both systems. The latter is a red herring, IMO, as discussed a bit
more inline with your hypothesis section.
- The most interesting things to latch on to so far, to me, are that we
have an ENOENT in __btrfs_free_extent on metadata block backrefs, and
that there is no thru-line wrt kernel version or raid setup.
>
> Possible analysis (could be wrong):
> * __btrfs_free_extent & btrfs_qgroup_cleanup_dropped_subvolume:
> Your btrfs-cleaner thread was in the middle of deleting an old snapshot. As it
> was freeing the blocks (extents), it attempted to update the quota accounting
> for that subvolume.
I don't see any evidence for that, as discussed above about the object
type referenced in the abort log. In fact, we don't really know that the
freeing even had to do with the subvolume being deleted as we were
running generic delayed refs as part of a consistency enforcing
transaction commit before digging into qgroup logic. We have not
connected the logical block that had the issue to subvol 83288, for
which we would probably need a tree dump.
> * failed to cleanup qgroup 0/83288: -2: The kernel tried to find the quota
> group record for the subvolume being deleted, but the record was missing
> or corrupted. To protect the filesystem, Btrfs panicked and locked
> itself read-only.
Unfortunately, this second bullet is nonsense, the qgroup cleanup log
is there simply because that is the caller of btrfs_commit_transaction
that consumed the failed delayed ref errno and also logged its own
failure. This is apparent from the stack trace and logs. This actually
confused and distracted me quite a bit :)
I'm generally a happy LLM user, but this sort of "jumping to
conclusions" behavior is annoyingly common if you don't take precautions
to be stern with them and demand that they go above and beyond to
justify their conclusions.
Thank you for your reports and all the extra details you have provided
so far. I hope we can get this figured out,
Boris
>
> ------------[ cut here ]------------
> BTRFS: Transaction aborted (error -2)
> WARNING: CPU: 7 PID: 2987 at fs/btrfs/extent-tree.c:2999 __btrfs_free_extent.isra.0+0xc41/0x1020 [btrfs
>
> Modules linked in: ftdi_sio usbserial ufs qnx4 hfsplus hfs cdrom minix msdos jfs nls_ucs2_utils xfs rpc
> v4 dns_resolver nfs netfs rfcomm xt_tcpudp snd_seq_dummy snd_hrtimer xt_conntrack nf_conntrack_netlink
> lgo xt_addrtype br_netfilter bridge stp llc ccm qrtr overlay cmac algif_hash algif_skcipher af_alg bnep
> JECT nf_reject_ipv4 xt_MASQUERADE xt_LOG nf_log_syslog nft_compat nft_chain_nat nf_nat nf_conntrack nf_
> efrag_ipv4 nf_tables binfmt_misc nls_ascii nls_cp437 vfat fat snd_soc_sof_sdw snd_soc_sdw_utils vboxdrv
> 11_sdca snd_soc_rt715_sdca snd_soc_rt1316_sdw dell_pc regmap_sdw_mbq platform_profile snd_hda_codec_int
> mic regmap_sdw kvm snd_hda_intel btusb snd_sof_pci_intel_tgl iwlmvm snd_sof_pci_intel_cnl btrtl irqbypa
> _hda_generic uvcvideo btintel soundwire_intel btbcm soundwire_generic_allocation videobuf2_vmalloc snd_
> w_bpt btmtk uvc videobuf2_memops mac80211 snd_sof_intel_hda_common
> videobuf2_v4l2 bluetooth snd_soc_hdac_hda videodev snd_sof_intel_hda_mlink videobuf2_common intel_uncore_frequency snd_sof_intel_hda libarc4 snd_hda_codec_hdmi snd_sof_pci intel_uncore_frequency_common mc ecdh_generic mei_hdcp soundwire_cadence snd_sof_xtensa_dsp mei_pxp ext4 x86_pkg_temp_thermal crc8 intel_rapl_msr processor_thermal_device_pci intel_powerclamp hid_sensor_als dell_laptop iwlwifi soundwire_bus processor_thermal_device dell_wmi hid_sensor_trigger processor_thermal_wt_hint rapl hid_sensor_iio_common platform_temperature_control snd_soc_avs industrialio_triggered_buffer crc16 snd_sof_probes processor_thermal_rfim intel_cstate iTCO_wdt mbcache dell_smbios kfifo_buf squashfs jbd2 loop dcdbas intel_uncore cfg80211 dell_smm_hwmon dell_wmi_sysman dell_wmi_ddv dell_wmi_descriptor firmware_attributes_class pcspkr snd_sof processor_thermal_rapl industrialio intel_pmc_bxt wmi_bmof snd_soc_hda_codec ucsi_acpi mei_me spd5118 iTCO_vendor_support intel_rapl_common snd_hda_ext_core snd_sof_utils typec_ucsi
> snd_intel_dspcfg watchdog mei processor_thermal_wt_req snd_intel_sdw_acpi rfkill intel_pmc_core typec processor_thermal_power_floor snd_soc_skl_hda_dsp igen6_edac processor_thermal_mbox roles int3403_thermal pmt_telemetry snd_soc_intel_sof_board_helpers int340x_thermal_zone snd_soc_acpi_intel_match pmt_discovery joydev snd_soc_acpi_intel_sdca_quirks pmt_class snd_soc_acpi int3400_thermal intel_hid intel_pmc_ssram_telemetry snd_soc_sdca acpi_thermal_rel sparse_keymap acpi_tad acpi_pad ac serio_raw snd_soc_core evdev snd_compress snd_pcm_dmaengine snd_soc_intel_hda_dsp_common snd_hda_codec snd_hda_core snd_hwdep snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_midi snd_seq_midi_event snd_seq snd_timer snd_rawmidi snd_seq_device snd_ctl_led snd soundcore ac97_bus coretemp nfsd msr ecryptfs auth_rpcgss nvme_fabrics nfs_acl lockd efi_pstore grace sunrpc nfnetlink ip_tables x_tables autofs4 crc32c_cryptoapi essiv authenc btrfs blake2b_generic dm_crypt dm_mod efivarfs raid10 raid456 async_raid6_recov async_memcpy
> async_pq async_xor async_tx xor raid6_pq raid1 raid0 md_mod sata_sil24 libata scsi_mod scsi_common e1000e r8169 realtek mdio_devres libphy mdio_bus mii xe configfs drm_gpusvm_helper drm_suballoc_helper hid_sensor_custom hid_sensor_hub intel_ishtp_hid nouveau mxm_wmi drm_gpuvm i915 gpu_sched drm_buddy drm_ttm_helper hid_multitouch ttm drm_exec i2c_algo_bit hid_generic drm_display_helper xhci_pci cec xhci_hcd rc_core nvme rtsx_pci_sdmmc i2c_hid_acpi intel_lpss_pci drm_client_lib i2c_hid nvme_core mmc_core intel_lpss usbcore video nvme_keyring i2c_i801 intel_ish_ipc drm_kms_helper hid ghash_clmulni_intel psmouse rtsx_pci thunderbolt intel_ishtp intel_vsec idma64 button usb_common i2c_smbus nvme_auth battery drm wmi aesni_intel
> CPU: 7 UID: 0 PID: 2987 Comm: btrfs-cleaner Tainted: G U OE 6.17.11+deb14-amd64 #1 PREEMPT(lazy) Debian 6.17.11-1
> Tainted: [U]=USER, [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
> Hardware name: Dell Inc. XPS 17 9730/0JP3YK, BIOS 1.23.0 09/04/2025
> RIP: 0010:__btrfs_free_extent.isra.0+0xc41/0x1020 [btrfs]
> Code: ff ff 48 c7 c7 d8 ac 98 c1 e8 ab 7a 6c e4 0f 0b c6 44 24 2f 01 e9 22 8a 0e 00 8b 74 24 10 48 c7 c7 d8 ac 98 c1 e8 8f 7a 6c e4 <0f> 0b e9 50 ff ff ff 48 8b 34 24 48 8b 76 60 48 89 74 24 08 48 8d
> RSP: 0000:ffffd372a10cfb40 EFLAGS: 00010246
> RAX: 0000000000000000 RBX: 00000124f8b48000 RCX: 0000000000000027
> RDX: ffff8d4d0f3dce48 RSI: 0000000000000001 RDI: ffff8d4d0f3dce40
> RBP: 0000000000004000 R08: 0000000000000000 R09: ffffd372a10cf9e0
> R10: ffff8d4d4f745068 R11: 00000000ffffdfff R12: 0000000000000000
> R13: 000000000000002b R14: 00000000000039c1 R15: ffff8d4b42451930
> FS: 0000000000000000(0000) GS:ffff8d4d669c8000(0000) knlGS:0000000000000000
> CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> CR2: 0000563efa2028b8 CR3: 0000000ce5c2c006 CR4: 0000000000f70ef0
> PKRU: 55555554
> Call Trace:
> <TASK>
> __btrfs_run_delayed_refs+0x2dc/0xf70 [btrfs]
> ? read_block_for_search+0x19e/0x400 [btrfs]
> ? set_extent_buffer_dirty+0x26/0x200 [btrfs]
> btrfs_run_delayed_refs+0x39/0x140 [btrfs]
> btrfs_commit_transaction+0x6d/0xdf0 [btrfs]
> btrfs_qgroup_cleanup_dropped_subvolume+0x49/0xb0 [btrfs]
> btrfs_drop_snapshot+0x78e/0xcc0 [btrfs]
> ? __pfx_cleaner_kthread+0x10/0x10 [btrfs]
> btrfs_clean_one_deleted_snapshot+0xc2/0x130 [btrfs]
> cleaner_kthread+0xdc/0x160 [btrfs]
> ? __pfx_cleaner_kthread+0x10/0x10 [btrfs]
> kthread+0xf9/0x240
> ? __pfx_kthread+0x10/0x10
> ret_from_fork+0x194/0x1c0
> ? __pfx_kthread+0x10/0x10
> ret_from_fork_asm+0x1a/0x30
> </TASK>
> ---[ end trace 0000000000000000 ]---
> BTRFS: error (device dm-4 state A) in do_free_extent_accounting:2999: errno=-2 No such entry
> BTRFS info (device dm-4 state EA): forced readonly
> BTRFS error (device dm-4 state EA): failed to run delayed ref for logical 1258303029248 num_bytes 16384 type 176 action 2 ref_mod 1: -2
> BTRFS: error (device dm-4 state EA) in btrfs_run_delayed_refs:2161: errno=-2 No such entry
> BTRFS warning (device dm-4 state EA): failed to cleanup qgroup 0/83288: -2
>
>
>
> On Fri, Apr 10, 2026 at 08:35:33PM -0700, Marc MERLIN wrote:
> >
> > It started with:
> > [23345.326321] BTRFS: error (device dm-0 state A) in do_free_extent_accounting:2996: errno=-2 No such entry
> > [23345.336394] BTRFS error (device dm-0 state EA): failed to run delayed ref for logical 15506102321152 num_bytes 16384 type 182 action 2 ref_mod 1: -2
> > [23345.350299] BTRFS: error (device dm-0 state EA) in btrfs_run_delayed_refs:2215: errno=-2 No such entry
> > [23345.360154] BTRFS warning (device dm-0 state EA):
> >
> > I ended up with:
> >
> > moremagic:~# mount -t btrfs -o rw,skip_balance,space_cache=v2,clear_cache /dev/mapper/crypt_bcache0 /mnt/btrfs_bigbackup
> > BTRFS: device label DS6 devid 1 transid 296950 /dev/mapper/crypt_bcache0 (251:0) scanned by mount (6029)
> > BTRFS info (device dm-0): first mount of filesystem a97dec85-a0d5-42ab-a0ef-e9b7479fbe43
> > BTRFS info (device dm-0): using crc32c (crc32c-generic) checksum algorithm
> > BTRFS warning (device dm-0): read-write for sector size 4096 with page size 16384 is experimental
> > BTRFS info (device dm-0): bdev /dev/mapper/crypt_bcache0 errs: wr 0, rd 0, flush 0, corrupt 5074, gen 0
> > ------------[ cut here ]------------
> > BTRFS: Transaction aborted (error -2)
> > WARNING: CPU: 3 PID: 6029 at fs/btrfs/extent-tree.c:2996 __btrfs_free_extent.isra.0+0x13a0/0x14a0 [btrfs]
> > Modules linked in: dm_crypt dm_mod bcache raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xt_MASQUERADE ipt_REJECT nf_reject_ipv4 xt_tcpudp xt_conntrack xt_LOG nf_log_syslog nft_compat nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nf_tables rfcomm algif_hash algif_skcipher af_alg bnep cp210x brcmfmac_wcc binfmt_misc usbserial hci_uart brcmfmac btbcm vc4 snd_soc_hdmi_codec brcmutil bluetooth drm_display_helper cfg80211 cec drm_dma_helper rpi_hevc_dec ecdh_generic v4l2_mem2mem ecc snd_soc_core pisp_be videobuf2_dma_contig v3d videobuf2_memops videobuf2_v4l2 gpu_sched rfkill videodev drm_shmem_helper snd_compress snd_pcm_dmaengine snd_pcm videobuf2_common rp1_pio snd_timer snd drm_kms_helper mc raspberrypi_gpiomem rp1_fw sg sch_fq_codel ecryptfs fuse drm drm_panel_orientation_quirks backlight nfnetlink ip_tables x_tables raid1 aes_ce_blk aes_ce_cipher ghash_ce gf128mul libaes sha2_ce spidev sha256_arm64 sha1_ce raspberrypi_hwmon sha1_generic ahci i2c_brcmstb spi_bcm2835
> > md_mod gpio_keys libahci pwm_fan rp1_adc libata rp1_mailbox nvmem_rmem uio_pdrv_genirq uio btrfs blake2b_generic xor xor_neon raid6_pq zram lz4_compress ipv6
> > CPU: 3 UID: 0 PID: 6029 Comm: mount Not tainted 6.12.47+rpt-rpi-2712 #1 Debian 1:6.12.47-1+rpt1
> > Hardware name: Raspberry Pi 5 Model B Rev 1.1 (DT)
> > pstate: 60400009 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
> > pc : __btrfs_free_extent.isra.0+0x13a0/0x14a0 [btrfs]
> > lr : __btrfs_free_extent.isra.0+0x13a0/0x14a0 [btrfs]
> > sp : ffffc000868bb680
> > x29: ffffc000868bb720 x28: 0000000000000000 x27: 0000000000002f02
> > x26: 000000000000007f x25: ffff8001de833aa0 x24: 0000000000004000
> > x23: 0000000000000000 x22: ffff800102b64e70 x21: 0000000000004000
> > x20: 00000e1a4bb88000 x19: 00000000fffffffe x18: 0000000000000000
> > x17: 0000000000000000 x16: 0000000000000000 x15: 0000000000000000
> > x14: 0000000000000000 x13: 0000000000000000 x12: 0000000000000000
> > x11: 00000000000000c0 x10: 0000000000001a40 x9 : ffffd06fce4e06c0
> > x8 : ffff80011f56e0a0 x7 : 000000042f72a7bd x6 : 0000000000000039
> > x5 : 0000000000000001 x4 : 0000000000001ab0 x3 : 0000000000000804
> > x2 : 0000000000000000 x1 : 0000000000000000 x0 : ffff80011f56c600
> > Call trace:
> > __btrfs_free_extent.isra.0+0x13a0/0x14a0 [btrfs]
> > __btrfs_run_delayed_refs+0x508/0xec0 [btrfs]
> > btrfs_run_delayed_refs+0x48/0x198 [btrfs]
> > btrfs_commit_transaction+0x88/0xe20 [btrfs]
> > btrfs_recover_relocation+0x55c/0x5d0 [btrfs]
> > btrfs_start_pre_rw_mount+0x1d4/0x470 [btrfs]
> > open_ctree+0x101c/0x13b8 [btrfs]
> > btrfs_get_tree+0x5b4/0x800 [btrfs]
> > vfs_get_tree+0x30/0x108
> > fc_mount+0x20/0x68
> > btrfs_get_tree+0x238/0x800 [btrfs]
> > vfs_get_tree+0x30/0x108
> > vfs_cmd_create+0x58/0xf8
> > __arm64_sys_fsconfig+0x444/0x5b8
> > invoke_syscall+0x50/0x120
> > el0_svc_common.constprop.0+0x48/0xf0
> > do_el0_svc+0x24/0x38
> > el0_svc+0x30/0xf8
> > el0t_64_sync_handler+0x120/0x130
> > el0t_64_sync+0x190/0x198
> > ---[ end trace 0000000000000000 ]---
> > BTRFS: error (device dm-0 state A) in do_free_extent_accounting:2996: errno=-2 No such entry
> > BTRFS error (device dm-0 state EA): failed to run delayed ref for logical 15506102321152 num_bytes 16384 type 182 action 2 ref_mod 1: -2
> > BTRFS: error (device dm-0 state EA) in btrfs_run_delayed_refs:2215: errno=-2 No such entry
> > BTRFS warning (device dm-0 state EA): failed to recover relocation: -2
> > BTRFS error (device dm-0 state EA): commit super ret -30
> > BTRFS error (device dm-0 state EA): open_ctree failed: -2
> >
> >
> > Gemini said
> >
> > The Btrfs "Ghost" Accounting When you added discard=async to your fstab
> > (or remounted with it), you told the Btrfs kernel module to start a specific
> > background thread.
> > Btrfs's Perspective: "The user told me to use async discard. I will now start a
> > list of every extent we delete so I can 'trim' them later in the background."
> > The Problem: Btrfs doesn't check if the underlying dm-crypt device actually
> > supports discards before it starts its own internal accounting.
> > The Result: Btrfs started tracking a massive list of "extents to be discarded"
> > in its memory and metadata.
> >
> > 2. The "No Such Entry" (-2) Race Condition The crash didn't happen because a
> > command hit a drive; it happened because of a logic race inside the kernel's
> > Btrfs code:
> > The Balance Thread: You were running a balance. This thread moves data from "Old
> > Block A" to "New Block B."
> > The Discard Thread: Because discard=async was on, the discard thread saw "Old
> > Block A" get freed. It put "Old Block A" on its "to-do list."
> > The Metadata Conflict: The balance thread finished moving the data and
> > successfully deleted the reference to "Old Block A" from the extent tree.
> > The Crash: A few milliseconds later, the async discard thread woke up and tried
> > to "pin" or "process" the metadata for "Old Block A." It looked in the tree,
> > found nothing (because the balance already deleted it), and threw an ENOENT
> > (Error -2: No such entry).
> > Btrfs panicked: "Wait, I was told to discard this block, but it doesn't exist in
> > my records anymore! Something is inconsistent!" → Transaction Abort.
> >
> > more details:
> > backuproot didn't work (read write)
> > I was forced to run
> > btrfstune --convert-from-block-group-tree /dev/mapper/crypt_bcache0
> > because
> > When you ran btrfs check --clear-space-cache v2, the tool did exactly
> > what it was supposed to do: it deleted the Free Space Tree and removed
> > the FREE_SPACE_TREE flag from your superblock.
> > The Conflict: Your 23TB array was formatted with the modern
> > block-group-tree feature (which speeds up mounting).
> > The Kernel Rule: The Btrfs kernel code explicitly dictates: If the Block
> > Group Tree is enabled, the Free Space Tree MUST also be enabled. * The
> > Crash: Because the FREE_SPACE_TREE flag is now missing, the kernel sees
> > an "illegal" superblock state and throws a fatal -22 error, refusing to
> > proceed to the mount options.
> >
> > This was vexing, hours lost removing the block group tree.
> > and when it was finally finished,
> > mount -t btrfs -o skip_balance /dev/mapper/crypt_bcache0 /mnt/btrfs_bigbackup/
> > did run, but crashed as above
> >
> > Now doing a repair in case it can salvage things.
> >
> > Marc
> > --
> > "A mouse is a device used to point at the xterm you want to type in" - A.S.R.
> >
> > Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
> >
>
> --
> "A mouse is a device used to point at the xterm you want to type in" - A.S.R.
>
> Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: Simple quota unsafe? RIP: 0010:__btrfs_free_extent.isra.0+0xc41/0x1020 [btrfs] / do_free_extent_accounting:2999: errno=-2 No such entry
2026-04-13 18:47 ` Boris Burkov
@ 2026-04-13 19:40 ` Marc MERLIN
2026-04-13 19:40 ` Marc MERLIN
2026-04-15 5:21 ` Marc MERLIN
0 siblings, 2 replies; 43+ messages in thread
From: Marc MERLIN @ 2026-04-13 19:40 UTC (permalink / raw)
To: Boris Burkov
Cc: linux-btrfs, Josef Bacik, QuWenruo, Qu Wenruo, Filipe Manana,
Chris Murphy, Zygo Blaxell, Roman Mamedov, Su Yue, Su Yue
On Mon, Apr 13, 2026 at 11:47:31AM -0700, Boris Burkov wrote:
> I am currently a little confused about your full story, so please help
> me make sure I understand. I would like to fix any squotas problems you
> are seeing if possible. I'm going to restate what I have understood from
> your reports to try to confirm I am following properly.
Sure thing, thanks for caring and replying.
For moremagic, the first report, I'm very close to wiping the filesystem
and starting over since I can't mount it read/write.
Ironically if I do that it would be a good time to turn on squota but
at the same time, it may not be safe in 6.12.
> I will call this report 1. Report 1 is from a rpi running 6.12 with possible out
> of tree modules and raid5.
Correct. moremagic is running Raspberry Pi debian which I understand is
running its own kernel to support the chips on the board. Sadly it means
I'm stuck at 6.12 for that one. I didn't know it would be an issue for
btrfs, but if you feel squotas are not ready/safe in 6.12, I'll disable
them (well, it looks like I will be doing that no matter what since I
can't have moremagic crash its 22TB filesystem , but still your feedback
will be valued).
> I'll call this report 2. Report 2 is from a laptop with no fancy raid
> and upstream kernel 6.17.
Correct. amd4 system and Package: linux-image-6.17.11+deb14-amd64-unsigned
from upstream debian which I assume is clean.
> Is that all accurate?
>
> Some further questions/observations:
> - I noticed that your paste from report 1 (https://pastebin.com/7HmQwy3n)
> had 16k pages and 4k block size:
> 2026-04-10T10:43:22.673638-07:00 moremagic kernel: BTRFS warning (device dm-0): read-write for sector size 4096 with page size 16384 is experimental
Yeah, I saw that too. I don't have much of a choice on arm, they have
switched to 16k.
I tried formatting my new filesystems as 16k native and then had to
revert once I realized it broke btrfs send/receive (cannot send from 4k
FS to 16k fs).
> which seems a bit risky on an old kernel. There were a lot of fixes for
> subpage block size support in recent kernels. I believe it has been
> quite stable for us on 6.16 but Qu can give the most authoritative
> answer on when that got solid.
I would love to know, yes.
> - Is the laptop also running subpage block size? Do you have a full
> dmesg from that systewhich you can share?
The laptop is as simple and basic
> - On which of these systems did you enable squotas and when?
5 systems, I enabled them 4 days ago and already had 2 crashes
I did also enable block-group-tree at the same time since I read
it really helps when I have 100+ snapshots on a single filesystem
(due to backup server, btrfs send receive and historical snapshots)
- rPi5 with that 6.12 kernel (moremagic, the one with the crash). One
crashed on a 4k btrfs disk array that was built a long time ago and I
just converted to squota
- 2nd Rpi5 with same kernel (no choice) with FS I just rebuilt last
night once I realized I can't use 16k pages. On top of raid5 on top of
SSDs. It's currently doing a multi day long btrfs receive to populate
aragorn:/mnt/btrfs_pool1# df -h | grep btrfs_pool
/dev/mapper/dshelf1 30T 3.4T 26T 12% /mnt/btrfs_pool1 (raid5 8 SSDs)
/dev/mapper/dshelf2 25T 84G 25T 1% /mnt/btrfs_pool2 (raid6 10 SSDs)
This one is fresh so the squotas would be useful. I have not disabled
them yet, but probably will as soon as you confirm it's probably not
safe, especially with 16k pages in the kernel but 4k in the filesystem
It's the only one I still have running now.
- laptop #1 where I enabled them on every filesystem (6.16.8-amd64, will
reboot to 6.19.11 as soon as I can reboot), but given that squotas are
kind of useless on existing filesystems since you can't backfill
missing quota data, I'm going to disable them now, I can't have that
laptop crash
- laptop #2 (merlin, the one that crashed). Similar debian install and
btrfs filesystems. btrfs_pool3 gets btrfs receive backups ,
juggles snapshots and btrfs balance nightly. Thankfully the
filesystem was fixable, I've brought it back online and disabled
quotas on all filesystems too.
- old file server running 6.16.8-amd64-preempt-sysrq-20241007 I built
myself from source. It also has not crashed yet, but I had just
enabled quotas on a big 22T spinning rust array that I'm finishing
a bit send/received to. Given that I really don't want to lose that
only backup left with the one I just lost on moremagic, and squota
isn't that useful on an existing filesystem, I just turned that one
off too. I ran for 2 days, but there were no nightly snapshots or
btrfs balance happening nightly on it.
> I don't see any evidence for that, as discussed above about the object
> type referenced in the abort log. In fact, we don't really know that the
> freeing even had to do with the subvolume being deleted as we were
> running generic delayed refs as part of a consistency enforcing
> transaction commit before digging into qgroup logic. We have not
> connected the logical block that had the issue to subvol 83288, for
> which we would probably need a tree dump.
understood.
> Unfortunately, this second bullet is nonsense, the qgroup cleanup log
> is there simply because that is the caller of btrfs_commit_transaction
> that consumed the failed delayed ref errno and also logged its own
> failure. This is apparent from the stack trace and logs. This actually
> confused and distracted me quite a bit :)
apologies for that. Normally I only use gemini on stuff I personally
know and understand, so I can easily tell if it's full of crap, but in
the case of btrfs kernel code, I'm clueless, sadly.
Despite that, hopefully the multiple oops and tracebacks give some clue.
For now, I've disabled squota everywhere but one after my 2nd laptop got
hit overnight. I'll see if I get more issues on it later which won't
prove anything, but give some correlation ("crashed with squota after 3
days, fine for many days without"). I understand the rPi is more
problematic due to non standard kernel I can't upgrade.
Thanks much for your time,
Marc
--
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
^ permalink raw reply [flat|nested] 43+ messages in thread* Re: Simple quota unsafe? RIP: 0010:__btrfs_free_extent.isra.0+0xc41/0x1020 [btrfs] / do_free_extent_accounting:2999: errno=-2 No such entry
2026-04-13 19:40 ` Marc MERLIN
@ 2026-04-13 19:40 ` Marc MERLIN
2026-04-15 5:21 ` Marc MERLIN
1 sibling, 0 replies; 43+ messages in thread
From: Marc MERLIN @ 2026-04-13 19:40 UTC (permalink / raw)
To: Boris Burkov
Cc: linux-btrfs, Josef Bacik, QuWenruo, Qu Wenruo, Filipe Manana,
Chris Murphy, Zygo Blaxell, Roman Mamedov, Su Yue, Su Yue
On Mon, Apr 13, 2026 at 11:47:31AM -0700, Boris Burkov wrote:
> I am currently a little confused about your full story, so please help
> me make sure I understand. I would like to fix any squotas problems you
> are seeing if possible. I'm going to restate what I have understood from
> your reports to try to confirm I am following properly.
Sure thing, thanks for caring and replying.
For moremagic, the first report, I'm very close to wiping the filesystem
and starting over since I can't mount it read/write.
Ironically if I do that it would be a good time to turn on squota but
at the same time, it may not be safe in 6.12.
> I will call this report 1. Report 1 is from a rpi running 6.12 with possible out
> of tree modules and raid5.
Correct. moremagic is running Raspberry Pi debian which I understand is
running its own kernel to support the chips on the board. Sadly it means
I'm stuck at 6.12 for that one. I didn't know it would be an issue for
btrfs, but if you feel squotas are not ready/safe in 6.12, I'll disable
them (well, it looks like I will be doing that no matter what since I
can't have moremagic crash its 22TB filesystem , but still your feedback
will be valued).
> I'll call this report 2. Report 2 is from a laptop with no fancy raid
> and upstream kernel 6.17.
Correct. amd4 system and Package: linux-image-6.17.11+deb14-amd64-unsigned
from upstream debian which I assume is clean.
> Is that all accurate?
>
> Some further questions/observations:
> - I noticed that your paste from report 1 (https://pastebin.com/7HmQwy3n)
> had 16k pages and 4k block size:
> 2026-04-10T10:43:22.673638-07:00 moremagic kernel: BTRFS warning (device dm-0): read-write for sector size 4096 with page size 16384 is experimental
Yeah, I saw that too. I don't have much of a choice on arm, they have
switched to 16k.
I tried formatting my new filesystems as 16k native and then had to
revert once I realized it broke btrfs send/receive (cannot send from 4k
FS to 16k fs).
> which seems a bit risky on an old kernel. There were a lot of fixes for
> subpage block size support in recent kernels. I believe it has been
> quite stable for us on 6.16 but Qu can give the most authoritative
> answer on when that got solid.
I would love to know, yes.
> - Is the laptop also running subpage block size? Do you have a full
> dmesg from that systewhich you can share?
The laptop is as simple and basic
> - On which of these systems did you enable squotas and when?
5 systems, I enabled them 4 days ago and already had 2 crashes
I did also enable block-group-tree at the same time since I read
it really helps when I have 100+ snapshots on a single filesystem
(due to backup server, btrfs send receive and historical snapshots)
- rPi5 with that 6.12 kernel (moremagic, the one with the crash). One
crashed on a 4k btrfs disk array that was built a long time ago and I
just converted to squota
- 2nd Rpi5 with same kernel (no choice) with FS I just rebuilt last
night once I realized I can't use 16k pages. On top of raid5 on top of
SSDs. It's currently doing a multi day long btrfs receive to populate
aragorn:/mnt/btrfs_pool1# df -h | grep btrfs_pool
/dev/mapper/dshelf1 30T 3.4T 26T 12% /mnt/btrfs_pool1 (raid5 8 SSDs)
/dev/mapper/dshelf2 25T 84G 25T 1% /mnt/btrfs_pool2 (raid6 10 SSDs)
This one is fresh so the squotas would be useful. I have not disabled
them yet, but probably will as soon as you confirm it's probably not
safe, especially with 16k pages in the kernel but 4k in the filesystem
It's the only one I still have running now.
- laptop #1 where I enabled them on every filesystem (6.16.8-amd64, will
reboot to 6.19.11 as soon as I can reboot), but given that squotas are
kind of useless on existing filesystems since you can't backfill
missing quota data, I'm going to disable them now, I can't have that
laptop crash
- laptop #2 (merlin, the one that crashed). Similar debian install and
btrfs filesystems. btrfs_pool3 gets btrfs receive backups ,
juggles snapshots and btrfs balance nightly. Thankfully the
filesystem was fixable, I've brought it back online and disabled
quotas on all filesystems too.
- old file server running 6.16.8-amd64-preempt-sysrq-20241007 I built
myself from source. It also has not crashed yet, but I had just
enabled quotas on a big 22T spinning rust array that I'm finishing
a bit send/received to. Given that I really don't want to lose that
only backup left with the one I just lost on moremagic, and squota
isn't that useful on an existing filesystem, I just turned that one
off too. I ran for 2 days, but there were no nightly snapshots or
btrfs balance happening nightly on it.
> I don't see any evidence for that, as discussed above about the object
> type referenced in the abort log. In fact, we don't really know that the
> freeing even had to do with the subvolume being deleted as we were
> running generic delayed refs as part of a consistency enforcing
> transaction commit before digging into qgroup logic. We have not
> connected the logical block that had the issue to subvol 83288, for
> which we would probably need a tree dump.
understood.
> Unfortunately, this second bullet is nonsense, the qgroup cleanup log
> is there simply because that is the caller of btrfs_commit_transaction
> that consumed the failed delayed ref errno and also logged its own
> failure. This is apparent from the stack trace and logs. This actually
> confused and distracted me quite a bit :)
apologies for that. Normally I only use gemini on stuff I personally
know and understand, so I can easily tell if it's full of crap, but in
the case of btrfs kernel code, I'm clueless, sadly.
Despite that, hopefully the multiple oops and tracebacks give some clue.
For now, I've disabled squota everywhere but one after my 2nd laptop got
hit overnight. I'll see if I get more issues on it later which won't
prove anything, but give some correlation ("crashed with squota after 3
days, fine for many days without"). I understand the rPi is more
problematic due to non standard kernel I can't upgrade.
Thanks much for your time,
Marc
--
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
^ permalink raw reply [flat|nested] 43+ messages in thread* Re: Simple quota unsafe? RIP: 0010:__btrfs_free_extent.isra.0+0xc41/0x1020 [btrfs] / do_free_extent_accounting:2999: errno=-2 No such entry
2026-04-13 19:40 ` Marc MERLIN
2026-04-13 19:40 ` Marc MERLIN
@ 2026-04-15 5:21 ` Marc MERLIN
2026-04-15 17:05 ` Boris Burkov
1 sibling, 1 reply; 43+ messages in thread
From: Marc MERLIN @ 2026-04-15 5:21 UTC (permalink / raw)
To: Boris Burkov
Cc: linux-btrfs, Josef Bacik, QuWenruo, Qu Wenruo, Filipe Manana,
Chris Murphy, Zygo Blaxell, Roman Mamedov, Su Yue, Su Yue
If you don't mind my asking, and even if it's just a guess:
1) do you feel squotas are likely safe and I hit another bug?
2) any chance it can be block-group-tree instead? Are those
considered reasonably safe? (I enabled due to the number of
snapshots, 100+, and being told it's bad for perf to have that
many, which block-group-tree is supposed to fix)
3) anyone still taking bugs for btrfs check --repair since it
fixed nothing on an unmountable filesystem
4) anything useful to anyone in my broken filesystem which might
be easy to fix but not by me before I give up, delete it all
and do the multi week restore?
Thanks,
Marc
--
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: Simple quota unsafe? RIP: 0010:__btrfs_free_extent.isra.0+0xc41/0x1020 [btrfs] / do_free_extent_accounting:2999: errno=-2 No such entry
2026-04-15 5:21 ` Marc MERLIN
@ 2026-04-15 17:05 ` Boris Burkov
2026-04-15 17:59 ` Marc MERLIN
0 siblings, 1 reply; 43+ messages in thread
From: Boris Burkov @ 2026-04-15 17:05 UTC (permalink / raw)
To: Marc MERLIN
Cc: linux-btrfs, Josef Bacik, QuWenruo, Qu Wenruo, Filipe Manana,
Chris Murphy, Zygo Blaxell, Roman Mamedov, Su Yue, Su Yue
On Tue, Apr 14, 2026 at 10:21:55PM -0700, Marc MERLIN wrote:
> If you don't mind my asking, and even if it's just a guess:
>
> 1) do you feel squotas are likely safe and I hit another bug?
I actually think it is a squota bug and am trying to reproduce it. I
think there may be a gap if we are deleting the qgroup in the same
transaction as the last delayed ref in the qgroup.
In meta production I have a carryover patch ignoring such enoents, which
makes me super suspicious.
I have not reproduced it yet, though.
The fact that the two stacks are in balance and qgroup deletion make me
suspicious. On the other hand, the fact that the second stack is in the
commit *before* the majority of the cleanup makes me less
suspicious.
> 2) any chance it can be block-group-tree instead? Are those
> considered reasonably safe? (I enabled due to the number of
> snapshots, 100+, and being told it's bad for perf to have that
> many, which block-group-tree is supposed to fix)
I think that is relatively unlikely.
> 3) anyone still taking bugs for btrfs check --repair since it
> fixed nothing on an unmountable filesystem
If my suspicion is correct and this does end up being a missing qgroup I
think that I could likely have a useful fix for check --repair too.
> 4) anything useful to anyone in my broken filesystem which might
> be easy to fix but not by me before I give up, delete it all
> and do the multi week restore?
I think trying to dump the state of the tree on your laptop could be
quite useful.
1258303029248 is the logical address we were freeing so its backrefs in
the extent tree will be interesting. As will whatever subvol tree it is
in, as will the qgroup tree entry for that subvolid. If you can share a
full dump then I can find all that myself.
ditto for the logical address that broke on the rpi, but if that is a
gigantic fs, maybe doing the dump will be pretty unwieldy..
Thanks for your help with this,
Boris
>
> Thanks,
> Marc
> --
> "A mouse is a device used to point at the xterm you want to type in" - A.S.R.
>
> Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: Simple quota unsafe? RIP: 0010:__btrfs_free_extent.isra.0+0xc41/0x1020 [btrfs] / do_free_extent_accounting:2999: errno=-2 No such entry
2026-04-15 17:05 ` Boris Burkov
@ 2026-04-15 17:59 ` Marc MERLIN
2026-04-15 18:44 ` Boris Burkov
0 siblings, 1 reply; 43+ messages in thread
From: Marc MERLIN @ 2026-04-15 17:59 UTC (permalink / raw)
To: Boris Burkov
Cc: linux-btrfs, Josef Bacik, QuWenruo, Qu Wenruo, Filipe Manana,
Chris Murphy, Zygo Blaxell, Roman Mamedov, Su Yue
On Wed, Apr 15, 2026 at 10:05:10AM -0700, Boris Burkov wrote:
> 1258303029248 is the logical address we were freeing so its backrefs in
> the extent tree will be interesting. As will whatever subvol tree it is
> in, as will the qgroup tree entry for that subvolid. If you can share a
> full dump then I can find all that myself.
Note that I fixed the laptop, thankfully didn't have to rebuild this
filesystem.
Sorry, not very good with the debug tools. Was that the right command?
merlin:/mnt/btrfs_pool3# btrfs inspect-internal dump-tree -t 1258303029248 /dev/mapper/pool3
btrfs-progs v6.14
total bytes 4074040004608
bytes used 2415294197760
uuid 49741c78-3949-4d7d-b77a-160ce071dee0
merlin:/mnt/btrfs_pool3#
> ditto for the logical address that broke on the rpi, but if that is a
> gigantic fs, maybe doing the dump will be pretty unwieldy..
That's the bigger one indeed, still waiting read only until I'm told I
can wipe it, or should wait for debug data request. It's been too long
already and I had spare drives, so I started a full restore on a new
set of drives and will swap them in for the current one I can't mount
read/write but can keep it around for debugging / potential future
check --repair patch
Thanks,
Marc
--
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: Simple quota unsafe? RIP: 0010:__btrfs_free_extent.isra.0+0xc41/0x1020 [btrfs] / do_free_extent_accounting:2999: errno=-2 No such entry
2026-04-15 17:59 ` Marc MERLIN
@ 2026-04-15 18:44 ` Boris Burkov
2026-04-15 20:22 ` Marc MERLIN
0 siblings, 1 reply; 43+ messages in thread
From: Boris Burkov @ 2026-04-15 18:44 UTC (permalink / raw)
To: Marc MERLIN
Cc: linux-btrfs, Josef Bacik, QuWenruo, Qu Wenruo, Filipe Manana,
Chris Murphy, Zygo Blaxell, Roman Mamedov, Su Yue
On Wed, Apr 15, 2026 at 10:59:44AM -0700, Marc MERLIN wrote:
> On Wed, Apr 15, 2026 at 10:05:10AM -0700, Boris Burkov wrote:
> > 1258303029248 is the logical address we were freeing so its backrefs in
> > the extent tree will be interesting. As will whatever subvol tree it is
> > in, as will the qgroup tree entry for that subvolid. If you can share a
> > full dump then I can find all that myself.
>
> Note that I fixed the laptop, thankfully didn't have to rebuild this
> filesystem.
>
> Sorry, not very good with the debug tools. Was that the right command?
> merlin:/mnt/btrfs_pool3# btrfs inspect-internal dump-tree -t 1258303029248 /dev/mapper/pool3
> btrfs-progs v6.14
> total bytes 4074040004608
> bytes used 2415294197760
> uuid 49741c78-3949-4d7d-b77a-160ce071dee0
> merlin:/mnt/btrfs_pool3#
>
>
For a giant fs where we don't want to dump all the metadata or it would
be difficult to share the resulting gigabytes of text, I would still get
quite a bit of benefit from:
btrfs inspect-internal dump-tree -t 2 /dev/tst/lol
and
btrfs inspect-internal dump-tree -t 8 /dev/tst/lol
which will dump the extent trees and quota trees respectively. The other
useful thing is to find the extent in question in the extent tree and
see which subvols refer to it via the backrefs and dump those trees. But
that is relatively less important. If the extent tree dumps way too
much, then doing a grep -C 50 for 15506102321152 would likely be enough
to show us the relevant backrefs.
> > ditto for the logical address that broke on the rpi, but if that is a
> > gigantic fs, maybe doing the dump will be pretty unwieldy..
>
> That's the bigger one indeed, still waiting read only until I'm told I
> can wipe it, or should wait for debug data request. It's been too long
> already and I had spare drives, so I started a full restore on a new
> set of drives and will swap them in for the current one I can't mount
> read/write but can keep it around for debugging / potential future
> check --repair patch
If there is no rush to wipe it, then let's keep it around until I rule
out whether a repair patch makes sense? Sorry if I misunderstood the
state. If there *is* a rush to wipe it, then let's confirm we got what
we needed out of tree dumping first, if possible.
>
> Thanks,
> Marc
> --
> "A mouse is a device used to point at the xterm you want to type in" - A.S.R.
>
> Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: Simple quota unsafe? RIP: 0010:__btrfs_free_extent.isra.0+0xc41/0x1020 [btrfs] / do_free_extent_accounting:2999: errno=-2 No such entry
2026-04-15 18:44 ` Boris Burkov
@ 2026-04-15 20:22 ` Marc MERLIN
2026-04-15 22:36 ` Boris Burkov
0 siblings, 1 reply; 43+ messages in thread
From: Marc MERLIN @ 2026-04-15 20:22 UTC (permalink / raw)
To: Boris Burkov
Cc: linux-btrfs, Josef Bacik, QuWenruo, Qu Wenruo, Filipe Manana,
Chris Murphy, Zygo Blaxell, Roman Mamedov
On Wed, Apr 15, 2026 at 11:44:27AM -0700, Boris Burkov wrote:
> For a giant fs where we don't want to dump all the metadata or it would
> be difficult to share the resulting gigabytes of text, I would still get
> quite a bit of benefit from:
>
> btrfs inspect-internal dump-tree -t 2 /dev/tst/lol
1.2G /tmp/tree2.txt
65M /tmp/tree2.txt.xz
do you want it?
> and
> btrfs inspect-internal dump-tree -t 8 /dev/tst/lol
btrfs-progs v6.14
total bytes 4074040004608
bytes used 2417457491968
uuid 49741c78-3949-4d7d-b77a-160ce071dee0
merlin:/mnt/btrfs_pool3#
> that is relatively less important. If the extent tree dumps way too
> much, then doing a grep -C 50 for 15506102321152 would likely be enough
> to show us the relevant backrefs.
No match:
merlin:/mnt/btrfs_pool3# btrfs inspect-internal dump-tree -t 2 /dev/mapper/pool3 | grep -50 15506102321152
merlin:/mnt/btrfs_pool3#
Anything else I can do?
> If there is no rush to wipe it, then let's keep it around until I rule
> out whether a repair patch makes sense? Sorry if I misunderstood the
> state. If there *is* a rush to wipe it, then let's confirm we got what
> we needed out of tree dumping first, if possible.
At this point, I can wait.
Thanks,
Marc
--
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: Simple quota unsafe? RIP: 0010:__btrfs_free_extent.isra.0+0xc41/0x1020 [btrfs] / do_free_extent_accounting:2999: errno=-2 No such entry
2026-04-15 20:22 ` Marc MERLIN
@ 2026-04-15 22:36 ` Boris Burkov
2026-04-15 22:55 ` Marc MERLIN
0 siblings, 1 reply; 43+ messages in thread
From: Boris Burkov @ 2026-04-15 22:36 UTC (permalink / raw)
To: Marc MERLIN
Cc: linux-btrfs, Josef Bacik, QuWenruo, Qu Wenruo, Filipe Manana,
Chris Murphy, Zygo Blaxell, Roman Mamedov
On Wed, Apr 15, 2026 at 01:22:42PM -0700, Marc MERLIN wrote:
> On Wed, Apr 15, 2026 at 11:44:27AM -0700, Boris Burkov wrote:
> > For a giant fs where we don't want to dump all the metadata or it would
> > be difficult to share the resulting gigabytes of text, I would still get
> > quite a bit of benefit from:
> >
> > btrfs inspect-internal dump-tree -t 2 /dev/tst/lol
>
> 1.2G /tmp/tree2.txt
> 65M /tmp/tree2.txt.xz
> do you want it?
Yes, please!
> > and
> > btrfs inspect-internal dump-tree -t 8 /dev/tst/lol
>
> btrfs-progs v6.14
> total bytes 4074040004608
> bytes used 2417457491968
> uuid 49741c78-3949-4d7d-b77a-160ce071dee0
> merlin:/mnt/btrfs_pool3#
>
So no output? That is surprising. Did you already disable quotas on this
filesystem?
> > that is relatively less important. If the extent tree dumps way too
> > much, then doing a grep -C 50 for 15506102321152 would likely be enough
> > to show us the relevant backrefs.
>
> No match:
> merlin:/mnt/btrfs_pool3# btrfs inspect-internal dump-tree -t 2 /dev/mapper/pool3 | grep -50 15506102321152
> merlin:/mnt/btrfs_pool3#
>
> Anything else I can do?
>
This is making me suspect *not* quotas now. That is the extent the mount
failed on. So it does seem to be the extent which is missing not the
qgroup.
However, it is possible that whatever has happened to the system since
the original abort / mount attempts is distorting our observations.
> > If there is no rush to wipe it, then let's keep it around until I rule
> > out whether a repair patch makes sense? Sorry if I misunderstood the
> > state. If there *is* a rush to wipe it, then let's confirm we got what
> > we needed out of tree dumping first, if possible.
>
> At this point, I can wait.
>
> Thanks,
> Marc
> --
> "A mouse is a device used to point at the xterm you want to type in" - A.S.R.
>
> Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: Simple quota unsafe? RIP: 0010:__btrfs_free_extent.isra.0+0xc41/0x1020 [btrfs] / do_free_extent_accounting:2999: errno=-2 No such entry
2026-04-15 22:36 ` Boris Burkov
@ 2026-04-15 22:55 ` Marc MERLIN
2026-04-15 23:25 ` Boris Burkov
2026-04-16 0:45 ` Boris Burkov
0 siblings, 2 replies; 43+ messages in thread
From: Marc MERLIN @ 2026-04-15 22:55 UTC (permalink / raw)
To: Boris Burkov
Cc: linux-btrfs, Josef Bacik, QuWenruo, Qu Wenruo, Filipe Manana,
Chris Murphy, Zygo Blaxell, Roman Mamedov
On Wed, Apr 15, 2026 at 03:36:29PM -0700, Boris Burkov wrote:
> On Wed, Apr 15, 2026 at 01:22:42PM -0700, Marc MERLIN wrote:
> > On Wed, Apr 15, 2026 at 11:44:27AM -0700, Boris Burkov wrote:
> > > For a giant fs where we don't want to dump all the metadata or it would
> > > be difficult to share the resulting gigabytes of text, I would still get
> > > quite a bit of benefit from:
> > >
> > > btrfs inspect-internal dump-tree -t 2 /dev/tst/lol
> >
> > 1.2G /tmp/tree2.txt
> > 65M /tmp/tree2.txt.xz
> > do you want it?
>
> Yes, please!
https://marc.merlins.org/tmp/tree2.txt.xz
please tell me when I can delete.
> > > btrfs inspect-internal dump-tree -t 8 /dev/tst/lol
> >
> > btrfs-progs v6.14
> > total bytes 4074040004608
> > bytes used 2417457491968
> > uuid 49741c78-3949-4d7d-b77a-160ce071dee0
> > merlin:/mnt/btrfs_pool3#
>
> So no output? That is surprising. Did you already disable quotas on this
> filesystem?
Ah, sorry, yes. First rule of sysadmining: after an unexplained crash:
revert the last thing(s) you did. 2 crashes in 3 days on systems that
were stable for years right after I turned on simple quotas and block-group-tree
could not be a coincidence.
Thankfully the laptop FS was not put into an unusable state and I was
able to get it back online quickly. As soon as I did, turned quotas off.
Since then I have upgraded the laptop to 6.19.11 and turned squotas back
on on that single filesystem.
Sorry if I removed evidence that could have helped. It's still on
moremagic, the rPi, since I could never remount read/write and turn off
quotas. Obviously I can do dump trees on it (please provide exact
commands you'd like so I don't mess them up)
I think there is also a way to backup the entire filesystem structure,
mangle the filenames, and omit the data to alllow a copy that is much
smaller and restore in a VM with spare blocks to emulate the FS without
needing 22TB
Let me know if either helps
Thanks,
Marc
--
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: Simple quota unsafe? RIP: 0010:__btrfs_free_extent.isra.0+0xc41/0x1020 [btrfs] / do_free_extent_accounting:2999: errno=-2 No such entry
2026-04-15 22:55 ` Marc MERLIN
@ 2026-04-15 23:25 ` Boris Burkov
2026-04-16 0:55 ` Marc MERLIN
2026-04-16 0:45 ` Boris Burkov
1 sibling, 1 reply; 43+ messages in thread
From: Boris Burkov @ 2026-04-15 23:25 UTC (permalink / raw)
To: Marc MERLIN
Cc: linux-btrfs, Josef Bacik, QuWenruo, Qu Wenruo, Filipe Manana,
Chris Murphy, Zygo Blaxell, Roman Mamedov
On Wed, Apr 15, 2026 at 03:55:39PM -0700, Marc MERLIN wrote:
> On Wed, Apr 15, 2026 at 03:36:29PM -0700, Boris Burkov wrote:
> > On Wed, Apr 15, 2026 at 01:22:42PM -0700, Marc MERLIN wrote:
> > > On Wed, Apr 15, 2026 at 11:44:27AM -0700, Boris Burkov wrote:
> > > > For a giant fs where we don't want to dump all the metadata or it would
> > > > be difficult to share the resulting gigabytes of text, I would still get
> > > > quite a bit of benefit from:
> > > >
> > > > btrfs inspect-internal dump-tree -t 2 /dev/tst/lol
> > >
> > > 1.2G /tmp/tree2.txt
> > > 65M /tmp/tree2.txt.xz
> > > do you want it?
> >
> > Yes, please!
>
> https://marc.merlins.org/tmp/tree2.txt.xz
> please tell me when I can delete.
>
Which machine is this from?
> > > > btrfs inspect-internal dump-tree -t 8 /dev/tst/lol
> > >
> > > btrfs-progs v6.14
> > > total bytes 4074040004608
> > > bytes used 2417457491968
> > > uuid 49741c78-3949-4d7d-b77a-160ce071dee0
> > > merlin:/mnt/btrfs_pool3#
> >
> > So no output? That is surprising. Did you already disable quotas on this
> > filesystem?
>
> Ah, sorry, yes. First rule of sysadmining: after an unexplained crash:
> revert the last thing(s) you did. 2 crashes in 3 days on systems that
> were stable for years right after I turned on simple quotas and block-group-tree
> could not be a coincidence.
> Thankfully the laptop FS was not put into an unusable state and I was
> able to get it back online quickly. As soon as I did, turned quotas off.
OK, the laptop is no longer useful for me to debug, which is fine, but
let's just disregard it going forward unless you have logs or other
dumps from before.
>
> Since then I have upgraded the laptop to 6.19.11 and turned squotas back
> on on that single filesystem.
If you do see the issue again there, please do capture the extent and
quota trees and I will look straight away.
>
> Sorry if I removed evidence that could have helped. It's still on
> moremagic, the rPi, since I could never remount read/write and turn off
> quotas. Obviously I can do dump trees on it (please provide exact
> commands you'd like so I don't mess them up)
No problem at all, they're your systems :)
My request for that extent tree + grep command was aimed at the rpi, I
believe. That is the system which produced this dmesg log, right?
https://pastebin.com/7HmQwy3n
If so, and if you ran the commands on the laptop where you already
wiped squotas, then that was just a miscommunication. If all that is
true, can you please run:
btrfs inspect-internal dump-tree -t 8 <dev>
and
btrfs inspect-internal dump-tree -t 2 <dev> | grep -50 15506102321152
on the rpi?
> I think there is also a way to backup the entire filesystem structure,
> mangle the filenames, and omit the data to alllow a copy that is much
> smaller and restore in a VM with spare blocks to emulate the FS without
> needing 22TB
>
> Let me know if either helps
>
> Thanks,
> Marc
> --
> "A mouse is a device used to point at the xterm you want to type in" - A.S.R.
>
> Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: Simple quota unsafe? RIP: 0010:__btrfs_free_extent.isra.0+0xc41/0x1020 [btrfs] / do_free_extent_accounting:2999: errno=-2 No such entry
2026-04-15 23:25 ` Boris Burkov
@ 2026-04-16 0:55 ` Marc MERLIN
2026-04-16 1:22 ` Boris Burkov
0 siblings, 1 reply; 43+ messages in thread
From: Marc MERLIN @ 2026-04-16 0:55 UTC (permalink / raw)
To: Boris Burkov
Cc: linux-btrfs, Josef Bacik, QuWenruo, Qu Wenruo, Filipe Manana,
Chris Murphy, Zygo Blaxell, Roman Mamedov
[-- Attachment #1: Type: text/plain, Size: 2029 bytes --]
On Wed, Apr 15, 2026 at 04:25:02PM -0700, Boris Burkov wrote:
> On Wed, Apr 15, 2026 at 03:55:39PM -0700, Marc MERLIN wrote:
> > On Wed, Apr 15, 2026 at 03:36:29PM -0700, Boris Burkov wrote:
> > > On Wed, Apr 15, 2026 at 01:22:42PM -0700, Marc MERLIN wrote:
> > > > On Wed, Apr 15, 2026 at 11:44:27AM -0700, Boris Burkov wrote:
> > > > > For a giant fs where we don't want to dump all the metadata or it would
> > > > > be difficult to share the resulting gigabytes of text, I would still get
> > > > > quite a bit of benefit from:
> > > > >
> > > > > btrfs inspect-internal dump-tree -t 2 /dev/tst/lol
> > > >
> > > > 1.2G /tmp/tree2.txt
> > > > 65M /tmp/tree2.txt.xz
> > > > do you want it?
> > >
> > > Yes, please!
> >
> > https://marc.merlins.org/tmp/tree2.txt.xz
> > please tell me when I can delete.
>
> Which machine is this from?
The laptop where quotas were removed after the filesystem turned R/O but
I was able to recover
> OK, the laptop is no longer useful for me to debug, which is fine, but
> let's just disregard it going forward unless you have logs or other
> dumps from before.
I do not. So I can delete the tree above?
> If you do see the issue again there, please do capture the extent and
> quota trees and I will look straight away.
Understood.
> My request for that extent tree + grep command was aimed at the rpi, I
> believe. That is the system which produced this dmesg log, right?
> https://pastebin.com/7HmQwy3n
Correct, that's the rPi with 6.12.xx
> If so, and if you ran the commands on the laptop where you already
> wiped squotas, then that was just a miscommunication. If all that is
> true, can you please run:
>
> btrfs inspect-internal dump-tree -t 8 <dev>
> and
> btrfs inspect-internal dump-tree -t 2 <dev> | grep -50 15506102321152
> on the rpi?
Sure thing, out2 and out8 attached.
Thanks,
Marc
--
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
[-- Attachment #2: out2.xz --]
[-- Type: application/x-xz, Size: 19980 bytes --]
[-- Attachment #3: out8.xz --]
[-- Type: application/x-xz, Size: 1472 bytes --]
^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: Simple quota unsafe? RIP: 0010:__btrfs_free_extent.isra.0+0xc41/0x1020 [btrfs] / do_free_extent_accounting:2999: errno=-2 No such entry
2026-04-16 0:55 ` Marc MERLIN
@ 2026-04-16 1:22 ` Boris Burkov
0 siblings, 0 replies; 43+ messages in thread
From: Boris Burkov @ 2026-04-16 1:22 UTC (permalink / raw)
To: Marc MERLIN
Cc: linux-btrfs, Josef Bacik, QuWenruo, Qu Wenruo, Filipe Manana,
Chris Murphy, Zygo Blaxell, Roman Mamedov
On Wed, Apr 15, 2026 at 05:55:55PM -0700, Marc MERLIN wrote:
> On Wed, Apr 15, 2026 at 04:25:02PM -0700, Boris Burkov wrote:
> > On Wed, Apr 15, 2026 at 03:55:39PM -0700, Marc MERLIN wrote:
> > > On Wed, Apr 15, 2026 at 03:36:29PM -0700, Boris Burkov wrote:
> > > > On Wed, Apr 15, 2026 at 01:22:42PM -0700, Marc MERLIN wrote:
> > > > > On Wed, Apr 15, 2026 at 11:44:27AM -0700, Boris Burkov wrote:
> > > > > > For a giant fs where we don't want to dump all the metadata or it would
> > > > > > be difficult to share the resulting gigabytes of text, I would still get
> > > > > > quite a bit of benefit from:
> > > > > >
> > > > > > btrfs inspect-internal dump-tree -t 2 /dev/tst/lol
> > > > >
> > > > > 1.2G /tmp/tree2.txt
> > > > > 65M /tmp/tree2.txt.xz
> > > > > do you want it?
> > > >
> > > > Yes, please!
> > >
> > > https://marc.merlins.org/tmp/tree2.txt.xz
> > > please tell me when I can delete.
> >
> > Which machine is this from?
>
> The laptop where quotas were removed after the filesystem turned R/O but
> I was able to recover
>
> > OK, the laptop is no longer useful for me to debug, which is fine, but
> > let's just disregard it going forward unless you have logs or other
> > dumps from before.
>
> I do not. So I can delete the tree above?
>
Go ahead, thanks.
> > If you do see the issue again there, please do capture the extent and
> > quota trees and I will look straight away.
>
> Understood.
>
> > My request for that extent tree + grep command was aimed at the rpi, I
> > believe. That is the system which produced this dmesg log, right?
> > https://pastebin.com/7HmQwy3n
>
> Correct, that's the rPi with 6.12.xx
>
> > If so, and if you ran the commands on the laptop where you already
> > wiped squotas, then that was just a miscommunication. If all that is
> > true, can you please run:
> >
> > btrfs inspect-internal dump-tree -t 8 <dev>
> > and
> > btrfs inspect-internal dump-tree -t 2 <dev> | grep -50 15506102321152
> > on the rpi?
>
> Sure thing, out2 and out8 attached.
>
Sweet, thank you.
> Thanks,
> Marc
> --
> "A mouse is a device used to point at the xterm you want to type in" - A.S.R.
>
> Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: Simple quota unsafe? RIP: 0010:__btrfs_free_extent.isra.0+0xc41/0x1020 [btrfs] / do_free_extent_accounting:2999: errno=-2 No such entry
2026-04-15 22:55 ` Marc MERLIN
2026-04-15 23:25 ` Boris Burkov
@ 2026-04-16 0:45 ` Boris Burkov
2026-04-16 1:08 ` Marc MERLIN
1 sibling, 1 reply; 43+ messages in thread
From: Boris Burkov @ 2026-04-16 0:45 UTC (permalink / raw)
To: Marc MERLIN
Cc: linux-btrfs, Josef Bacik, QuWenruo, Qu Wenruo, Filipe Manana,
Chris Murphy, Zygo Blaxell, Roman Mamedov
On Wed, Apr 15, 2026 at 03:55:39PM -0700, Marc MERLIN wrote:
> On Wed, Apr 15, 2026 at 03:36:29PM -0700, Boris Burkov wrote:
> > On Wed, Apr 15, 2026 at 01:22:42PM -0700, Marc MERLIN wrote:
> > > On Wed, Apr 15, 2026 at 11:44:27AM -0700, Boris Burkov wrote:
> > > > For a giant fs where we don't want to dump all the metadata or it would
> > > > be difficult to share the resulting gigabytes of text, I would still get
> > > > quite a bit of benefit from:
> > > >
> > > > btrfs inspect-internal dump-tree -t 2 /dev/tst/lol
> > >
> > > 1.2G /tmp/tree2.txt
> > > 65M /tmp/tree2.txt.xz
> > > do you want it?
> >
> > Yes, please!
>
> https://marc.merlins.org/tmp/tree2.txt.xz
> please tell me when I can delete.
>
> > > > btrfs inspect-internal dump-tree -t 8 /dev/tst/lol
> > >
> > > btrfs-progs v6.14
> > > total bytes 4074040004608
> > > bytes used 2417457491968
> > > uuid 49741c78-3949-4d7d-b77a-160ce071dee0
> > > merlin:/mnt/btrfs_pool3#
> >
> > So no output? That is surprising. Did you already disable quotas on this
> > filesystem?
>
> Ah, sorry, yes. First rule of sysadmining: after an unexplained crash:
> revert the last thing(s) you did. 2 crashes in 3 days on systems that
> were stable for years right after I turned on simple quotas and block-group-tree
> could not be a coincidence.
I am 99% sure this is a squotas bug now. I am able to reproduce friends
of your issue like:
create subvol
enable squota (usage is 0)
delete subvol qgroup (allowed, usage is 0)
add an extent to subvol (BOOM)
and also
enable squota
create subvol (usage = 0 until delayed refs run)
destroy subvol qgroup (usage 0)
sync (actually create usage, happens to swallow enoent)
delete subvol
sync (BOOM)
I haven't reproduced one yet that hits on subvol cleanup / extent
deletion without manually deleting a qgroup, but it is probably possible
as well. Did you happen to do any manual subvol qgroup deletion at any
point?
I think some variant of only logging that enoent will greatly harden
things and when we are deleting it really should not matter.
I also think "btrfstune --remove-simple-quota" may fix your fs. It will
have to do work in O(extents), and might fail in its fake mount, though.
In terms of a check option, I think I can try to detect that we
have a owner_ref item for a missing subvol and then nuke it. Working on
that next.
Thanks,
Boris
> Thankfully the laptop FS was not put into an unusable state and I was
> able to get it back online quickly. As soon as I did, turned quotas off.
>
> Since then I have upgraded the laptop to 6.19.11 and turned squotas back
> on on that single filesystem.
>
> Sorry if I removed evidence that could have helped. It's still on
> moremagic, the rPi, since I could never remount read/write and turn off
> quotas. Obviously I can do dump trees on it (please provide exact
> commands you'd like so I don't mess them up)
> I think there is also a way to backup the entire filesystem structure,
> mangle the filenames, and omit the data to alllow a copy that is much
> smaller and restore in a VM with spare blocks to emulate the FS without
> needing 22TB
>
> Let me know if either helps
>
> Thanks,
> Marc
> --
> "A mouse is a device used to point at the xterm you want to type in" - A.S.R.
>
> Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: Simple quota unsafe? RIP: 0010:__btrfs_free_extent.isra.0+0xc41/0x1020 [btrfs] / do_free_extent_accounting:2999: errno=-2 No such entry
2026-04-16 0:45 ` Boris Burkov
@ 2026-04-16 1:08 ` Marc MERLIN
2026-04-16 1:25 ` Boris Burkov
0 siblings, 1 reply; 43+ messages in thread
From: Marc MERLIN @ 2026-04-16 1:08 UTC (permalink / raw)
To: Boris Burkov; +Cc: linux-btrfs
On Wed, Apr 15, 2026 at 05:45:52PM -0700, Boris Burkov wrote:
> I am 99% sure this is a squotas bug now. I am able to reproduce friends
> of your issue like:
Great news, thanks or looking into it.
> I also think "btrfstune --remove-simple-quota" may fix your fs. It will
> have to do work in O(extents), and might fail in its fake mount, though.
moremagic:/tmp# btrfstune --remove-simple-quota /dev/mapper/crypt_bcache0
bad eb member end: ptr 0x4000 start 15495725432832 member offset 16384 size 1
bad eb member end: ptr 0x4000 start 16568941559808 member offset 16384 size 1
bad eb member end: ptr 0x4000 start 16133011357696 member offset 16384 size 1
bad eb member end: ptr 0x4000 start 15495760642048 member offset 16384 size 1
bad eb member end: ptr 0x4000 start 15641229164544 member offset 16384 size 1
bad eb member end: ptr 0x4000 start 16027775762432 member offset 16384 size 1
bad eb member end: ptr 0x4000 start 16217851576320 member offset 16384 size 1
bad eb member end: ptr 0x4000 start 16217853050880 member offset 16384 size 1
bad eb member end: ptr 0x4000 start 15505960157184 member offset 16384 size 1
(...) many lines
It's still running after a while and giving hundreds of those lines.
Willl report back when it's done.
> In terms of a check option, I think I can try to detect that we
> have a owner_ref item for a missing subvol and then nuke it. Working on
> that next.
thank you. But I can only check that if btrfs tune doesn't fix it. Did
you want me to stop the tune or let it run to completion?
Marc
--
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: Simple quota unsafe? RIP: 0010:__btrfs_free_extent.isra.0+0xc41/0x1020 [btrfs] / do_free_extent_accounting:2999: errno=-2 No such entry
2026-04-16 1:08 ` Marc MERLIN
@ 2026-04-16 1:25 ` Boris Burkov
2026-04-16 16:51 ` Simple quota unsafe (FIXED: btrfstune --remove-simple-quota worked) Marc MERLIN
2026-04-16 17:21 ` Simple quota unsafe? RIP: 0010:__btrfs_free_extent.isra.0+0xc41/0x1020 [btrfs] / do_free_extent_accounting:2999: errno=-2 No such entry Marc MERLIN
0 siblings, 2 replies; 43+ messages in thread
From: Boris Burkov @ 2026-04-16 1:25 UTC (permalink / raw)
To: Marc MERLIN; +Cc: linux-btrfs
On Wed, Apr 15, 2026 at 06:08:03PM -0700, Marc MERLIN wrote:
> On Wed, Apr 15, 2026 at 05:45:52PM -0700, Boris Burkov wrote:
> > I am 99% sure this is a squotas bug now. I am able to reproduce friends
> > of your issue like:
>
> Great news, thanks or looking into it.
>
> > I also think "btrfstune --remove-simple-quota" may fix your fs. It will
> > have to do work in O(extents), and might fail in its fake mount, though.
>
> moremagic:/tmp# btrfstune --remove-simple-quota /dev/mapper/crypt_bcache0
> bad eb member end: ptr 0x4000 start 15495725432832 member offset 16384 size 1
> bad eb member end: ptr 0x4000 start 16568941559808 member offset 16384 size 1
> bad eb member end: ptr 0x4000 start 16133011357696 member offset 16384 size 1
> bad eb member end: ptr 0x4000 start 15495760642048 member offset 16384 size 1
> bad eb member end: ptr 0x4000 start 15641229164544 member offset 16384 size 1
> bad eb member end: ptr 0x4000 start 16027775762432 member offset 16384 size 1
> bad eb member end: ptr 0x4000 start 16217851576320 member offset 16384 size 1
> bad eb member end: ptr 0x4000 start 16217853050880 member offset 16384 size 1
> bad eb member end: ptr 0x4000 start 15505960157184 member offset 16384 size 1
> (...) many lines
>
> It's still running after a while and giving hundreds of those lines.
>
> Willl report back when it's done.
>
Not too promising, but no theory what would cause that off the top of my
head. You can let it run, now that you started it, I think.
> > In terms of a check option, I think I can try to detect that we
> > have a owner_ref item for a missing subvol and then nuke it. Working on
> > that next.
>
> thank you. But I can only check that if btrfs tune doesn't fix it. Did
> you want me to stop the tune or let it run to completion?
>
That's ok, I think the condition to fix is easy enough to produce.
> Marc
> --
> "A mouse is a device used to point at the xterm you want to type in" - A.S.R.
>
> Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: Simple quota unsafe (FIXED: btrfstune --remove-simple-quota worked)
2026-04-16 1:25 ` Boris Burkov
@ 2026-04-16 16:51 ` Marc MERLIN
2026-04-16 17:21 ` Simple quota unsafe? RIP: 0010:__btrfs_free_extent.isra.0+0xc41/0x1020 [btrfs] / do_free_extent_accounting:2999: errno=-2 No such entry Marc MERLIN
1 sibling, 0 replies; 43+ messages in thread
From: Marc MERLIN @ 2026-04-16 16:51 UTC (permalink / raw)
To: Boris Burkov; +Cc: linux-btrfs
Ok, we have a winner!
> > > I also think "btrfstune --remove-simple-quota" may fix your fs. It will
> > > have to do work in O(extents), and might fail in its fake mount, though.
> >
> > moremagic:/tmp# btrfstune --remove-simple-quota /dev/mapper/crypt_bcache0
> > bad eb member end: ptr 0x4000 start 15495725432832 member offset 16384 size 1
> > bad eb member end: ptr 0x4000 start 16568941559808 member offset 16384 size 1
Took a while to remove the quota, but once it did:
moremagic:/tmp# mount -o rw,enospc_debug,skip_balance /dev/mapper/crypt_bcache0 /mnt/btrfs_bigbackup
moremagic:/tmp# cd /mnt/btrfs_bigbackup
moremagic:/mnt/btrfs_bigbackup# btrfs balance cancel `pwd`
BTRFS info (device dm-0): setting compat-ro feature flag for FREE_SPACE_TREE (0x1)
BTRFS info (device dm-0): setting compat-ro feature flag for FREE_SPACE_TREE_VALID (0x2)
BTRFS info (device dm-0): balance: resume skipped
BTRFS info (device dm-0): checking UUID tree
BTRFS info (device dm-0): enabling ssd optimizations
BTRFS info (device dm-0): enabling free space tree
BTRFS info (device dm-0): balance: canceled
And now I'm good. I can re-re-convert to block group trees (took hours
overnight to go back):
moremagic:~# btrfstune --convert-to-block-group-tree /dev/mapper/crypt_bcache0
Converted the filesystem to block group tree feature
moremagic:~# mount /dev/mapper/crypt_bcache0 /mnt/btrfs_bigbackup
BTRFS: device label DS6 devid 1 transid 298273 /dev/mapper/crypt_bcache0 (251:0) scanned by mount (1030419)
BTRFS info (device dm-0): first mount of filesystem a97dec85-a0d5-42ab-a0ef-e9b7479fbe43
BTRFS info (device dm-0): using crc32c (crc32c-generic) checksum algorithm
BTRFS info (device dm-0): forcing free space tree for sector size 4096 with page size 16384
BTRFS warning (device dm-0): read-write for sector size 4096 with page size 16384 is experimental
BTRFS info (device dm-0): bdev /dev/mapper/crypt_bcache0 errs: wr 0, rd 0, flush 0, corrupt 5074, gen 0
BTRFS info (device dm-0): checking UUID tree
BTRFS info (device dm-0): enabling ssd optimizations
BTRFS info (device dm-0): enabling free space tree
I'm back in business, thank you.
Will definitely keep squota off for this one. Thanks again about the
--remove-simple-quota suggestion, of all the things I tried, this is one
I hadn't.
Marc
--
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: Simple quota unsafe? RIP: 0010:__btrfs_free_extent.isra.0+0xc41/0x1020 [btrfs] / do_free_extent_accounting:2999: errno=-2 No such entry
2026-04-16 1:25 ` Boris Burkov
2026-04-16 16:51 ` Simple quota unsafe (FIXED: btrfstune --remove-simple-quota worked) Marc MERLIN
@ 2026-04-16 17:21 ` Marc MERLIN
2026-04-16 21:36 ` Boris Burkov
1 sibling, 1 reply; 43+ messages in thread
From: Marc MERLIN @ 2026-04-16 17:21 UTC (permalink / raw)
To: Boris Burkov; +Cc: linux-btrfs
No help needed, just showing it happened overnight with balance and squota in 6.19
on my laptop. It looks like squota and balance are simply not compatible until the
fix for squota, lands.
Now that I know more, I rescued the FS and removed squota on my own and it's good again.
It looks pretty clear that btrfs balance seems to trigger the bug in
squota:
[37930.185472] BTRFS info (device dm-3): balance: start -dusage=20
[37930.187010] BTRFS info (device dm-3): relocating block group 4502498312192 flags data
[37930.417388] BTRFS info (device dm-3): found 83 extents, stage: move data extents
[37930.896615] BTRFS info (device dm-3): found 83 extents, stage: update data pointers
[37931.311770] BTRFS info (device dm-3): relocating block group 4501424570368 flags data
[37931.355759] BTRFS info (device dm-3): found 11 extents, stage: move data extents
[37931.501741] BTRFS info (device dm-3): found 11 extents, stage: update data pointers
[37931.507435] BTRFS: Transaction aborted (error -2)
[37931.508762] BTRFS: error (device dm-3 state A) in do_free_extent_accounting:3002: errno=-2 No such entry
[37931.508766] BTRFS info (device dm-3 state EA): forced readonly
[37931.508769] BTRFS error (device dm-3 state EA): failed to run delayed ref for logical 907535646720 num_bytes 16384 type 182 action 2 ref_mod 1: -2
[37931.508773] BTRFS: error (device dm-3 state EA) in btrfs_run_delayed_refs:2166: errno=-2 No such entry
[37931.508798] BTRFS info (device dm-3 state EA): balance: ended with status: -2
[37931.507428] ------------[ cut here ]------------
[37931.507435] BTRFS: Transaction aborted (error -2)
[37931.507438] WARNING: fs/btrfs/extent-tree.c:3002 at __btrfs_free_extent.isra.0+0xdf6/0xfe0 [btrfs], CPU#0: btrfs/153883
[37931.507519] Modules linked in: rpcsec_gss_krb5 nfsv4 dns_resolver nfs netfs sd_mod sg uas usb_storage uinput nf_conntrack_netlink xfrm_user xfrm_algo xt_addrtype br_netfilter bridge stp llc xt_tcpudp xt_conntrack ccm rfcomm snd_seq_dummy snd_hrtimer overlay ipt_REJECT nf_reject_ipv4 xt_MASQUERADE xt_LOG nf_log_syslog nft_compat nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 qrtr nf_tables cmac algif_hash algif_skcipher af_alg bnep binfmt_misc uvcvideo btusb videobuf2_vmalloc btmtk uvc videobuf2_memops btrtl videobuf2_v4l2 btbcm btintel videodev videobuf2_common bluetooth mc ftdi_sio ecdh_generic usbserial nls_ascii nls_cp437 vfat fat ext4 crc16 mbcache squashfs jbd2 loop intel_uncore_frequency intel_uncore_frequency_common x86_pkg_temp_thermal iwlmvm intel_powerclamp snd_hda_codec_intelhdmi kvm_intel snd_hda_codec_alc269 snd_hda_scodec_component snd_hda_codec_realtek_lib mac80211 snd_hda_codec_generic kvm snd_soc_dmic rtsx_pci_sdmmc mmc_core libarc4 snd_sof_pci_intel_tgl irqbypass mei_hdcp mei_pxp
[37931.507593] intel_rapl_msr snd_sof_pci_intel_cnl rapl intel_cstate snd_sof_intel_hda_generic soundwire_intel snd_sof_pci snd_sof_xtensa_dsp snd_sof_intel_hda_sdw_bpt processor_thermal_device_pci_legacy iwlwifi intel_uncore think_lmi snd_sof_intel_hda_common iTCO_wdt snd_soc_sdw_utils processor_thermal_device processor_thermal_wt_hint intel_pmc_bxt snd_soc_hdac_hda firmware_attributes_class snd_hda_intel platform_temperature_control wmi_bmof iTCO_vendor_support snd_sof_intel_hda_mlink processor_thermal_soc_slider snd_soc_avs cfg80211 processor_thermal_rfim ee1004 snd_sof_intel_hda processor_thermal_rapl watchdog pcspkr snd_soc_hda_codec rtsx_pci snd_hda_codec_hdmi intel_rapl_common ucsi_acpi processor_thermal_wt_req typec_ucsi mei_me soundwire_cadence processor_thermal_power_floor mei snd_hda_ext_core typec soundwire_generic_allocation processor_thermal_mbox crc8 roles soundwire_bus thinkpad_acpi intel_soc_dts_iosf snd_sof_probes nvram platform_profile int3403_thermal snd_sof rfkill int340x_thermal_zone ac
[37931.507644] intel_pmc_core snd_sof_utils pmt_telemetry snd_intel_dspcfg pmt_discovery pmt_class snd_intel_sdw_acpi int3400_thermal joydev intel_hid intel_pmc_ssram_telemetry sparse_keymap snd_soc_skl_hda_dsp acpi_thermal_rel snd_soc_intel_sof_board_helpers acpi_pad button acpi_tad snd_soc_acpi_intel_match evdev snd_soc_acpi_intel_sdca_quirks snd_soc_acpi serio_raw snd_soc_sdca snd_soc_core snd_compress snd_pcm_dmaengine snd_soc_intel_hda_dsp_common snd_hda_codec snd_hda_core snd_hwdep snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_midi snd_seq_midi_event snd_seq snd_timer snd_rawmidi snd_seq_device snd_ctl_led snd nfsd soundcore ac97_bus coretemp auth_rpcgss msr ecryptfs nfs_acl lockd grace efi_pstore sunrpc nfnetlink ip_tables x_tables autofs4 crc32c_cryptoapi essiv authenc btrfs blake2b libblake2b dm_crypt dm_mod efivarfs raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq raid1 raid0 md_mod sata_sil24 libata scsi_mod scsi_common e1000e r8169 realtek mii xe intel_vsec drm_gpuvm
[37931.507715] configfs drm_gpusvm_helper gpu_sched drm_ttm_helper drm_exec drm_suballoc_helper hid_generic usbhid hid i915 drm_buddy ttm i2c_algo_bit drm_display_helper xhci_pci cec nvme xhci_hcd rc_core nvme_core drm_client_lib usbcore drm_kms_helper nvme_keyring video i2c_i801 ghash_clmulni_intel psmouse i2c_smbus nvme_auth igc hkdf drm thunderbolt usb_common battery wmi aesni_intel
[37931.507761] CPU: 0 UID: 0 PID: 153883 Comm: btrfs Tainted: G S U 6.19.11+deb14-amd64 #1 PREEMPT(lazy) Debian 6.19.11-1
[37931.507765] Tainted: [S]=CPU_OUT_OF_SPEC, [U]=USER
[37931.507767] Hardware name: LENOVO 20YU002JUS/20YU002JUS, BIOS N37ET61W (1.97 ) 09/17/2025
[37931.507769] RIP: 0010:__btrfs_free_extent.isra.0+0xdfa/0xfe0 [btrfs]
[37931.507830] Code: 48 c7 c6 50 e1 a0 c1 48 8b 78 60 e8 30 80 0f 00 e8 4b e9 62 d9 44 88 6c 24 2f e9 98 aa 0e 00 48 8d 3d 6a 49 f4 ff 8b 74 24 14 <67> 48 0f b9 3a e9 ef fd ff ff 48 8d 3d 65 49 f4 ff 8b 74 24 14 67
[37931.507833] RSP: 0018:ffffcc4b75297800 EFLAGS: 00010206
[37931.507836] RAX: 000000000000001c RBX: 000000d34d570000 RCX: ffff891a04502000
[37931.507838] RDX: 0000000000000000 RSI: 00000000fffffffe RDI: ffffffffc176e990
[37931.507840] RBP: 0000000000000000 R08: ffff89275fe8e2d0 R09: ffff89275fe8e2d0
[37931.507842] R10: ffff891a04502000 R11: 0000000000000011 R12: 0000000000004000
[37931.507843] R13: 0000000000000000 R14: ffff892b31377380 R15: 0000000000000010
[37931.507845] FS: 00007fa538c1d3c0(0000) GS:ffff893951da7000(0000) knlGS:0000000000000000
[37931.507847] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[37931.507849] CR2: 00003c840b875500 CR3: 00000010f962b005 CR4: 0000000000f70ef0
[37931.507851] PKRU: 55555554
[37931.507853] Call Trace:
[37931.507856] <TASK>
[37931.507862] __btrfs_run_delayed_refs+0x2e3/0xfb0 [btrfs]
[37931.507915] ? need_preemptive_reclaim+0x36/0x190 [btrfs]
[37931.507976] ? reserve_bytes+0x177/0x4a0 [btrfs]
[37931.508032] btrfs_run_delayed_refs+0x3b/0x120 [btrfs]
[37931.508085] btrfs_commit_transaction+0x6a/0xd50 [btrfs]
[37931.508142] ? btrfs_init_metadata_block_rsv+0x28/0x40 [btrfs]
[37931.508200] ? start_transaction+0x228/0x840 [btrfs]
[37931.508256] prepare_to_relocate+0x135/0x1d0 [btrfs]
[37931.508323] relocate_block_group+0x6b/0x530 [btrfs]
[37931.508382] btrfs_relocate_block_group+0x26e/0x3d0 [btrfs]
[37931.508439] btrfs_relocate_chunk+0x44/0x170 [btrfs]
[37931.508504] btrfs_balance+0xa5b/0x1640 [btrfs]
[37931.508568] btrfs_ioctl+0x28ac/0x2dd0 [btrfs]
[37931.508635] ? kmem_cache_free+0x508/0x550
[37931.508640] ? dput.part.0+0x27/0x110
[37931.508643] ? __x64_sys_close+0x3d/0x80
[37931.508647] __x64_sys_ioctl+0x97/0xe0
[37931.508651] do_syscall_64+0x81/0x5e0
[37931.508656] ? syscall_exit_work+0x143/0x1b0
[37931.508665] ? do_syscall_64+0xbe/0x5e0
[37931.508671] ? syscall_exit_work+0x143/0x1b0
[37931.508674] ? do_syscall_64+0xbe/0x5e0
[37931.508676] ? kernfs_fop_llseek+0x78/0xc0
[37931.508682] ? syscall_exit_work+0x143/0x1b0
[37931.508685] ? do_syscall_64+0xbe/0x5e0
[37931.508687] ? clear_bhb_loop+0x50/0xa0
[37931.508690] ? clear_bhb_loop+0x50/0xa0
[37931.508692] ? clear_bhb_loop+0x50/0xa0
[37931.508695] entry_SYSCALL_64_after_hwframe+0x76/0x7e
[37931.508697] RIP: 0033:0x7fa538d3ad3b
[37931.508741] Code: 00 48 89 44 24 18 31 c0 48 8d 44 24 60 c7 04 24 10 00 00 00 48 89 44 24 08 48 8d 44 24 20 48 89 44 24 10 b8 10 00 00 00 0f 05 <89> c2 3d 00 f0 ff ff 77 1c 48 8b 44 24 18 64 48 2b 04 25 28 00 00
[37931.508744] RSP: 002b:00007fff72a15140 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[37931.508747] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007fa538d3ad3b
[37931.508749] RDX: 00007fff72a15228 RSI: 00000000c4009420 RDI: 0000000000000003
[37931.508750] RBP: 0000000000000003 R08: 0000000000000000 R09: 0000000000000000
[37931.508752] R10: 0000000000000000 R11: 0000000000000246 R12: 00007fff72a15228
[37931.508753] R13: 00007fff72a15bd8 R14: 00007fff72a17da5 R15: 0000000000000001
[37931.508756] </TASK>
[37931.508758] ---[ end trace 0000000000000000 ]---
[37931.508762] BTRFS: error (device dm-3 state A) in do_free_extent_accounting:3002: errno=-2 No such entry
[37931.508766] BTRFS info (device dm-3 state EA): forced readonly
[37931.508769] BTRFS error (device dm-3 state EA): failed to run delayed ref for logical 907535646720 num_bytes 16384 type 182 action 2 ref_mod 1: -2
[37931.508773] BTRFS: error (device dm-3 state EA) in btrfs_run_delayed_refs:2166: errno=-2 No such entry
sauron:/boot# mount /dev/mapper/pool4 /mnt/btrfs_pool4
[65964.162801] BTRFS error (device dm-3 state EMA): remounting read-write after error is not allowed
For google and people who find that thread, the quicker recovery is:
sauron:/mnt/btrfs_pool3# btrfs rescue clear-ino-cache /dev/mapper/pool4
(...)
Successfully cleaned up ino cache for root id: 108518
Successfully cleared ino cache
sauron:/mnt/btrfs_pool3# mount -o rw,clear_cache,enospc_debug,skip_balance LABEL=btrfs_pool4 /mnt/btrfs_pool4
sauron:/mnt/btrfs_pool3# btrfs quota disable /mnt/btrfs_pool4
sauron:/mnt/btrfs_pool3# mount /dev/mapper/pool4 /mnt/btrfs_pool4
sauron:/mnt/btrfs_pool3# cd /mnt/btrfs_pool4
sauron:/mnt/btrfs_pool4# btrfs balance start -dusage=20 `pwd`
Done, had to relocate 0 out of 2008 chunks
sauron:/mnt/btrfs_pool4#
btrfs tune would have likely also worked to remove squota, but the trick
after that is mounting with skip_balance or otherwise, even after quota
removal, balance restarts right away and puts the FS read only again
if squota isn't disabled first.
Marc
--
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: Simple quota unsafe? RIP: 0010:__btrfs_free_extent.isra.0+0xc41/0x1020 [btrfs] / do_free_extent_accounting:2999: errno=-2 No such entry
2026-04-16 17:21 ` Simple quota unsafe? RIP: 0010:__btrfs_free_extent.isra.0+0xc41/0x1020 [btrfs] / do_free_extent_accounting:2999: errno=-2 No such entry Marc MERLIN
@ 2026-04-16 21:36 ` Boris Burkov
2026-04-16 21:47 ` Marc MERLIN
0 siblings, 1 reply; 43+ messages in thread
From: Boris Burkov @ 2026-04-16 21:36 UTC (permalink / raw)
To: Marc MERLIN; +Cc: linux-btrfs
On Thu, Apr 16, 2026 at 10:21:44AM -0700, Marc MERLIN wrote:
> No help needed, just showing it happened overnight with balance and squota in 6.19
> on my laptop. It looks like squota and balance are simply not compatible until the
> fix for squota, lands.
> Now that I know more, I rescued the FS and removed squota on my own and it's good again.
>
> It looks pretty clear that btrfs balance seems to trigger the bug in
> squota:
Thank you for the additional report, this is helpful. I am attempting to
reproduce directly with balance to be sure I've got everything.
One question just to be extra careful:
In all these situations, were you ever manually managing either the
subvolumes or the qgroups? e.g., when, if ever, where you deleting
qgroups and / or subvols?
Thanks,
Boris
> [37930.185472] BTRFS info (device dm-3): balance: start -dusage=20
> [37930.187010] BTRFS info (device dm-3): relocating block group 4502498312192 flags data
> [37930.417388] BTRFS info (device dm-3): found 83 extents, stage: move data extents
> [37930.896615] BTRFS info (device dm-3): found 83 extents, stage: update data pointers
> [37931.311770] BTRFS info (device dm-3): relocating block group 4501424570368 flags data
> [37931.355759] BTRFS info (device dm-3): found 11 extents, stage: move data extents
> [37931.501741] BTRFS info (device dm-3): found 11 extents, stage: update data pointers
> [37931.507435] BTRFS: Transaction aborted (error -2)
> [37931.508762] BTRFS: error (device dm-3 state A) in do_free_extent_accounting:3002: errno=-2 No such entry
> [37931.508766] BTRFS info (device dm-3 state EA): forced readonly
> [37931.508769] BTRFS error (device dm-3 state EA): failed to run delayed ref for logical 907535646720 num_bytes 16384 type 182 action 2 ref_mod 1: -2
> [37931.508773] BTRFS: error (device dm-3 state EA) in btrfs_run_delayed_refs:2166: errno=-2 No such entry
> [37931.508798] BTRFS info (device dm-3 state EA): balance: ended with status: -2
>
> [37931.507428] ------------[ cut here ]------------
> [37931.507435] BTRFS: Transaction aborted (error -2)
> [37931.507438] WARNING: fs/btrfs/extent-tree.c:3002 at __btrfs_free_extent.isra.0+0xdf6/0xfe0 [btrfs], CPU#0: btrfs/153883
> [37931.507519] Modules linked in: rpcsec_gss_krb5 nfsv4 dns_resolver nfs netfs sd_mod sg uas usb_storage uinput nf_conntrack_netlink xfrm_user xfrm_algo xt_addrtype br_netfilter bridge stp llc xt_tcpudp xt_conntrack ccm rfcomm snd_seq_dummy snd_hrtimer overlay ipt_REJECT nf_reject_ipv4 xt_MASQUERADE xt_LOG nf_log_syslog nft_compat nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 qrtr nf_tables cmac algif_hash algif_skcipher af_alg bnep binfmt_misc uvcvideo btusb videobuf2_vmalloc btmtk uvc videobuf2_memops btrtl videobuf2_v4l2 btbcm btintel videodev videobuf2_common bluetooth mc ftdi_sio ecdh_generic usbserial nls_ascii nls_cp437 vfat fat ext4 crc16 mbcache squashfs jbd2 loop intel_uncore_frequency intel_uncore_frequency_common x86_pkg_temp_thermal iwlmvm intel_powerclamp snd_hda_codec_intelhdmi kvm_intel snd_hda_codec_alc269 snd_hda_scodec_component snd_hda_codec_realtek_lib mac80211 snd_hda_codec_generic kvm snd_soc_dmic rtsx_pci_sdmmc mmc_core libarc4 snd_sof_pci_intel_tgl irqbypass mei_hdcp mei_pxp
> [37931.507593] intel_rapl_msr snd_sof_pci_intel_cnl rapl intel_cstate snd_sof_intel_hda_generic soundwire_intel snd_sof_pci snd_sof_xtensa_dsp snd_sof_intel_hda_sdw_bpt processor_thermal_device_pci_legacy iwlwifi intel_uncore think_lmi snd_sof_intel_hda_common iTCO_wdt snd_soc_sdw_utils processor_thermal_device processor_thermal_wt_hint intel_pmc_bxt snd_soc_hdac_hda firmware_attributes_class snd_hda_intel platform_temperature_control wmi_bmof iTCO_vendor_support snd_sof_intel_hda_mlink processor_thermal_soc_slider snd_soc_avs cfg80211 processor_thermal_rfim ee1004 snd_sof_intel_hda processor_thermal_rapl watchdog pcspkr snd_soc_hda_codec rtsx_pci snd_hda_codec_hdmi intel_rapl_common ucsi_acpi processor_thermal_wt_req typec_ucsi mei_me soundwire_cadence processor_thermal_power_floor mei snd_hda_ext_core typec soundwire_generic_allocation processor_thermal_mbox crc8 roles soundwire_bus thinkpad_acpi intel_soc_dts_iosf snd_sof_probes nvram platform_profile int3403_thermal snd_sof rfkill int340x_thermal_zone ac
> [37931.507644] intel_pmc_core snd_sof_utils pmt_telemetry snd_intel_dspcfg pmt_discovery pmt_class snd_intel_sdw_acpi int3400_thermal joydev intel_hid intel_pmc_ssram_telemetry sparse_keymap snd_soc_skl_hda_dsp acpi_thermal_rel snd_soc_intel_sof_board_helpers acpi_pad button acpi_tad snd_soc_acpi_intel_match evdev snd_soc_acpi_intel_sdca_quirks snd_soc_acpi serio_raw snd_soc_sdca snd_soc_core snd_compress snd_pcm_dmaengine snd_soc_intel_hda_dsp_common snd_hda_codec snd_hda_core snd_hwdep snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_midi snd_seq_midi_event snd_seq snd_timer snd_rawmidi snd_seq_device snd_ctl_led snd nfsd soundcore ac97_bus coretemp auth_rpcgss msr ecryptfs nfs_acl lockd grace efi_pstore sunrpc nfnetlink ip_tables x_tables autofs4 crc32c_cryptoapi essiv authenc btrfs blake2b libblake2b dm_crypt dm_mod efivarfs raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq raid1 raid0 md_mod sata_sil24 libata scsi_mod scsi_common e1000e r8169 realtek mii xe intel_vsec drm_gpuvm
> [37931.507715] configfs drm_gpusvm_helper gpu_sched drm_ttm_helper drm_exec drm_suballoc_helper hid_generic usbhid hid i915 drm_buddy ttm i2c_algo_bit drm_display_helper xhci_pci cec nvme xhci_hcd rc_core nvme_core drm_client_lib usbcore drm_kms_helper nvme_keyring video i2c_i801 ghash_clmulni_intel psmouse i2c_smbus nvme_auth igc hkdf drm thunderbolt usb_common battery wmi aesni_intel
> [37931.507761] CPU: 0 UID: 0 PID: 153883 Comm: btrfs Tainted: G S U 6.19.11+deb14-amd64 #1 PREEMPT(lazy) Debian 6.19.11-1
> [37931.507765] Tainted: [S]=CPU_OUT_OF_SPEC, [U]=USER
> [37931.507767] Hardware name: LENOVO 20YU002JUS/20YU002JUS, BIOS N37ET61W (1.97 ) 09/17/2025
> [37931.507769] RIP: 0010:__btrfs_free_extent.isra.0+0xdfa/0xfe0 [btrfs]
> [37931.507830] Code: 48 c7 c6 50 e1 a0 c1 48 8b 78 60 e8 30 80 0f 00 e8 4b e9 62 d9 44 88 6c 24 2f e9 98 aa 0e 00 48 8d 3d 6a 49 f4 ff 8b 74 24 14 <67> 48 0f b9 3a e9 ef fd ff ff 48 8d 3d 65 49 f4 ff 8b 74 24 14 67
> [37931.507833] RSP: 0018:ffffcc4b75297800 EFLAGS: 00010206
> [37931.507836] RAX: 000000000000001c RBX: 000000d34d570000 RCX: ffff891a04502000
> [37931.507838] RDX: 0000000000000000 RSI: 00000000fffffffe RDI: ffffffffc176e990
> [37931.507840] RBP: 0000000000000000 R08: ffff89275fe8e2d0 R09: ffff89275fe8e2d0
> [37931.507842] R10: ffff891a04502000 R11: 0000000000000011 R12: 0000000000004000
> [37931.507843] R13: 0000000000000000 R14: ffff892b31377380 R15: 0000000000000010
> [37931.507845] FS: 00007fa538c1d3c0(0000) GS:ffff893951da7000(0000) knlGS:0000000000000000
> [37931.507847] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [37931.507849] CR2: 00003c840b875500 CR3: 00000010f962b005 CR4: 0000000000f70ef0
> [37931.507851] PKRU: 55555554
> [37931.507853] Call Trace:
> [37931.507856] <TASK>
> [37931.507862] __btrfs_run_delayed_refs+0x2e3/0xfb0 [btrfs]
> [37931.507915] ? need_preemptive_reclaim+0x36/0x190 [btrfs]
> [37931.507976] ? reserve_bytes+0x177/0x4a0 [btrfs]
> [37931.508032] btrfs_run_delayed_refs+0x3b/0x120 [btrfs]
> [37931.508085] btrfs_commit_transaction+0x6a/0xd50 [btrfs]
> [37931.508142] ? btrfs_init_metadata_block_rsv+0x28/0x40 [btrfs]
> [37931.508200] ? start_transaction+0x228/0x840 [btrfs]
> [37931.508256] prepare_to_relocate+0x135/0x1d0 [btrfs]
> [37931.508323] relocate_block_group+0x6b/0x530 [btrfs]
> [37931.508382] btrfs_relocate_block_group+0x26e/0x3d0 [btrfs]
> [37931.508439] btrfs_relocate_chunk+0x44/0x170 [btrfs]
> [37931.508504] btrfs_balance+0xa5b/0x1640 [btrfs]
> [37931.508568] btrfs_ioctl+0x28ac/0x2dd0 [btrfs]
> [37931.508635] ? kmem_cache_free+0x508/0x550
> [37931.508640] ? dput.part.0+0x27/0x110
> [37931.508643] ? __x64_sys_close+0x3d/0x80
> [37931.508647] __x64_sys_ioctl+0x97/0xe0
> [37931.508651] do_syscall_64+0x81/0x5e0
> [37931.508656] ? syscall_exit_work+0x143/0x1b0
> [37931.508665] ? do_syscall_64+0xbe/0x5e0
> [37931.508671] ? syscall_exit_work+0x143/0x1b0
> [37931.508674] ? do_syscall_64+0xbe/0x5e0
> [37931.508676] ? kernfs_fop_llseek+0x78/0xc0
> [37931.508682] ? syscall_exit_work+0x143/0x1b0
> [37931.508685] ? do_syscall_64+0xbe/0x5e0
> [37931.508687] ? clear_bhb_loop+0x50/0xa0
> [37931.508690] ? clear_bhb_loop+0x50/0xa0
> [37931.508692] ? clear_bhb_loop+0x50/0xa0
> [37931.508695] entry_SYSCALL_64_after_hwframe+0x76/0x7e
> [37931.508697] RIP: 0033:0x7fa538d3ad3b
> [37931.508741] Code: 00 48 89 44 24 18 31 c0 48 8d 44 24 60 c7 04 24 10 00 00 00 48 89 44 24 08 48 8d 44 24 20 48 89 44 24 10 b8 10 00 00 00 0f 05 <89> c2 3d 00 f0 ff ff 77 1c 48 8b 44 24 18 64 48 2b 04 25 28 00 00
> [37931.508744] RSP: 002b:00007fff72a15140 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
> [37931.508747] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007fa538d3ad3b
> [37931.508749] RDX: 00007fff72a15228 RSI: 00000000c4009420 RDI: 0000000000000003
> [37931.508750] RBP: 0000000000000003 R08: 0000000000000000 R09: 0000000000000000
> [37931.508752] R10: 0000000000000000 R11: 0000000000000246 R12: 00007fff72a15228
> [37931.508753] R13: 00007fff72a15bd8 R14: 00007fff72a17da5 R15: 0000000000000001
> [37931.508756] </TASK>
> [37931.508758] ---[ end trace 0000000000000000 ]---
> [37931.508762] BTRFS: error (device dm-3 state A) in do_free_extent_accounting:3002: errno=-2 No such entry
> [37931.508766] BTRFS info (device dm-3 state EA): forced readonly
> [37931.508769] BTRFS error (device dm-3 state EA): failed to run delayed ref for logical 907535646720 num_bytes 16384 type 182 action 2 ref_mod 1: -2
> [37931.508773] BTRFS: error (device dm-3 state EA) in btrfs_run_delayed_refs:2166: errno=-2 No such entry
>
> sauron:/boot# mount /dev/mapper/pool4 /mnt/btrfs_pool4
> [65964.162801] BTRFS error (device dm-3 state EMA): remounting read-write after error is not allowed
>
> For google and people who find that thread, the quicker recovery is:
> sauron:/mnt/btrfs_pool3# btrfs rescue clear-ino-cache /dev/mapper/pool4
> (...)
> Successfully cleaned up ino cache for root id: 108518
> Successfully cleared ino cache
> sauron:/mnt/btrfs_pool3# mount -o rw,clear_cache,enospc_debug,skip_balance LABEL=btrfs_pool4 /mnt/btrfs_pool4
> sauron:/mnt/btrfs_pool3# btrfs quota disable /mnt/btrfs_pool4
>
> sauron:/mnt/btrfs_pool3# mount /dev/mapper/pool4 /mnt/btrfs_pool4
> sauron:/mnt/btrfs_pool3# cd /mnt/btrfs_pool4
> sauron:/mnt/btrfs_pool4# btrfs balance start -dusage=20 `pwd`
> Done, had to relocate 0 out of 2008 chunks
> sauron:/mnt/btrfs_pool4#
>
> btrfs tune would have likely also worked to remove squota, but the trick
> after that is mounting with skip_balance or otherwise, even after quota
> removal, balance restarts right away and puts the FS read only again
> if squota isn't disabled first.
>
> Marc
> --
> "A mouse is a device used to point at the xterm you want to type in" - A.S.R.
>
> Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: Simple quota unsafe? RIP: 0010:__btrfs_free_extent.isra.0+0xc41/0x1020 [btrfs] / do_free_extent_accounting:2999: errno=-2 No such entry
2026-04-16 21:36 ` Boris Burkov
@ 2026-04-16 21:47 ` Marc MERLIN
2026-04-17 21:51 ` Boris Burkov
0 siblings, 1 reply; 43+ messages in thread
From: Marc MERLIN @ 2026-04-16 21:47 UTC (permalink / raw)
To: Boris Burkov; +Cc: linux-btrfs
On Thu, Apr 16, 2026 at 02:36:32PM -0700, Boris Burkov wrote:
> On Thu, Apr 16, 2026 at 10:21:44AM -0700, Marc MERLIN wrote:
> > No help needed, just showing it happened overnight with balance and squota in 6.19
> > on my laptop. It looks like squota and balance are simply not compatible until the
> > fix for squota, lands.
> > Now that I know more, I rescued the FS and removed squota on my own and it's good again.
> >
> > It looks pretty clear that btrfs balance seems to trigger the bug in
> > squota:
>
> Thank you for the additional report, this is helpful. I am attempting to
> reproduce directly with balance to be sure I've got everything.
>
> One question just to be extra careful:
> In all these situations, were you ever manually managing either the
> subvolumes or the qgroups? e.g., when, if ever, where you deleting
> qgroups and / or subvols?
All 3 filesystems this happened on were btrfs receive recipients, so
that created snapshots nightly (when it crashed each time).
Also after btrfs receive, my script deletes the oldest backup that rolls
out the rotation.
Then I also have some amount btrfs-snaps that makes hourly/daily/weekly
snapshots
So yes, multiple per day per subvolume multipled by many subvolume
(10-ish)
I have squotas still running on a single host but I turned off btrfs
balance nightly cronjob as well as btrfs scrub, and it has not crashed
yet (it's a 20TB+ backup recipient built from scratch, I woudl love for
squotas to work on it)
Marc
--
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: Simple quota unsafe? RIP: 0010:__btrfs_free_extent.isra.0+0xc41/0x1020 [btrfs] / do_free_extent_accounting:2999: errno=-2 No such entry
2026-04-16 21:47 ` Marc MERLIN
@ 2026-04-17 21:51 ` Boris Burkov
2026-04-17 22:37 ` Marc MERLIN
0 siblings, 1 reply; 43+ messages in thread
From: Boris Burkov @ 2026-04-17 21:51 UTC (permalink / raw)
To: Marc MERLIN; +Cc: linux-btrfs
On Thu, Apr 16, 2026 at 02:47:14PM -0700, Marc MERLIN wrote:
> On Thu, Apr 16, 2026 at 02:36:32PM -0700, Boris Burkov wrote:
> > On Thu, Apr 16, 2026 at 10:21:44AM -0700, Marc MERLIN wrote:
> > > No help needed, just showing it happened overnight with balance and squota in 6.19
> > > on my laptop. It looks like squota and balance are simply not compatible until the
> > > fix for squota, lands.
> > > Now that I know more, I rescued the FS and removed squota on my own and it's good again.
> > >
> > > It looks pretty clear that btrfs balance seems to trigger the bug in
> > > squota:
> >
> > Thank you for the additional report, this is helpful. I am attempting to
> > reproduce directly with balance to be sure I've got everything.
> >
> > One question just to be extra careful:
> > In all these situations, were you ever manually managing either the
> > subvolumes or the qgroups? e.g., when, if ever, where you deleting
> > qgroups and / or subvols?
>
> All 3 filesystems this happened on were btrfs receive recipients, so
> that created snapshots nightly (when it crashed each time).
> Also after btrfs receive, my script deletes the oldest backup that rolls
> out the rotation.
> Then I also have some amount btrfs-snaps that makes hourly/daily/weekly
> snapshots
>
> So yes, multiple per day per subvolume multipled by many subvolume
> (10-ish)
Just to be extra clear as it is relevant for fully understanding the
repro-space, are you managing these purely with receive / subvol delete
commands? Did you ever delete a qgroup that was straggling around or add
automation to do that (btrfs qgroup destroy <qgid>) or did you ever
disable/enable squotas besides the first time?
Were all the squota enables on existing fs-es?
>
> I have squotas still running on a single host but I turned off btrfs
> balance nightly cronjob as well as btrfs scrub, and it has not crashed
> yet (it's a 20TB+ backup recipient built from scratch, I woudl love for
> squotas to work on it)
>
> Marc
> --
> "A mouse is a device used to point at the xterm you want to type in" - A.S.R.
>
> Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: Simple quota unsafe? RIP: 0010:__btrfs_free_extent.isra.0+0xc41/0x1020 [btrfs] / do_free_extent_accounting:2999: errno=-2 No such entry
2026-04-17 21:51 ` Boris Burkov
@ 2026-04-17 22:37 ` Marc MERLIN
2026-04-17 23:16 ` Boris Burkov
0 siblings, 1 reply; 43+ messages in thread
From: Marc MERLIN @ 2026-04-17 22:37 UTC (permalink / raw)
To: Boris Burkov; +Cc: linux-btrfs
On Fri, Apr 17, 2026 at 02:51:27PM -0700, Boris Burkov wrote:
> Just to be extra clear as it is relevant for fully understanding the
> repro-space, are you managing these purely with receive / subvol delete
> commands?
yes
> Did you ever delete a qgroup that was straggling around or add
> automation to do that (btrfs qgroup destroy <qgid>) or did you ever
> disable/enable squotas besides the first time?
no to all 3
> Were all the squota enables on existing fs-es?
Yes. My understanding was that I have to create the FS first and then
turn on squota. If I could turn it on as an option to mkfs.btffs, I
would but I didn't find such an option.
Marc
--
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: Simple quota unsafe? RIP: 0010:__btrfs_free_extent.isra.0+0xc41/0x1020 [btrfs] / do_free_extent_accounting:2999: errno=-2 No such entry
2026-04-17 22:37 ` Marc MERLIN
@ 2026-04-17 23:16 ` Boris Burkov
2026-04-18 0:18 ` Marc MERLIN
0 siblings, 1 reply; 43+ messages in thread
From: Boris Burkov @ 2026-04-17 23:16 UTC (permalink / raw)
To: Marc MERLIN; +Cc: linux-btrfs
On Fri, Apr 17, 2026 at 03:37:54PM -0700, Marc MERLIN wrote:
> On Fri, Apr 17, 2026 at 02:51:27PM -0700, Boris Burkov wrote:
> > Just to be extra clear as it is relevant for fully understanding the
> > repro-space, are you managing these purely with receive / subvol delete
> > commands?
>
> yes
>
> > Did you ever delete a qgroup that was straggling around or add
> > automation to do that (btrfs qgroup destroy <qgid>) or did you ever
> > disable/enable squotas besides the first time?
>
> no to all 3
>
Rad, so there is some more "exciting" bug with balance lurking there.
> > Were all the squota enables on existing fs-es?
>
> Yes. My understanding was that I have to create the FS first and then
> turn on squota. If I could turn it on as an option to mkfs.btffs, I
> would but I didn't find such an option.
>
mkfs.btrfs -O squota <dev>
should make a fresh btrfs with squota enabled. But we can worry about
that once I get the fixes in. :)
> Marc
> --
> "A mouse is a device used to point at the xterm you want to type in" - A.S.R.
>
> Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: Simple quota unsafe? RIP: 0010:__btrfs_free_extent.isra.0+0xc41/0x1020 [btrfs] / do_free_extent_accounting:2999: errno=-2 No such entry
2026-04-17 23:16 ` Boris Burkov
@ 2026-04-18 0:18 ` Marc MERLIN
0 siblings, 0 replies; 43+ messages in thread
From: Marc MERLIN @ 2026-04-18 0:18 UTC (permalink / raw)
To: Boris Burkov; +Cc: linux-btrfs
On Fri, Apr 17, 2026 at 04:16:03PM -0700, Boris Burkov wrote:
> Rad, so there is some more "exciting" bug with balance lurking there.
Correct, all 3 times the bug happened, it was during balance.
And now that you mention it, all 3 filesystems it happened to had a
bunch of data already before I enabled squota on them (because those
FSes were several years old, and created with much older kernels).
On all 3, I ran:
btrfstune -n -x -r $DEV; btrfstune --enable-simple-quota $DEV ; btrfstune --convert-to-block-group-tree $DEV
and then on the mounted filesystem
btrfs quota enable --simple .
I'm not sure how many of -n -x -r were already enabled before, but
100% know squota were not and neither were block group trees
My last remaining FS with squota running is the only one that had
squotas on it from the start (with the btrfs quota comamand but the FS
was empty).
If you feel the bug might not trigger in that use case, I can try to
re-enable balance and scrub on it, but it's another 30TB FS, so it sucks
if I lose it. Then again, I think we now know I can recover without
losing it, so should I go ahead and re-enable them with 6.19 (even if
6.19 did crash on my older FS where I enabled squota after data was
already there?)
> mkfs.btrfs -O squota <dev>
Aaah, I was missing that option, but even if it's a make time, do I
still need to turn them on with "btrfs quota enable --simple mountpoint"?
If there is no good documentation on all this, it's been 12 years since
I wrote all those missing docs/howtos on btrfs in https://marc.merlins.org/perso/btrfs/
happy to make a new one to put a few new notes on squotas and block-group-tree
which I was unaware of until just a week ago.
On the plus side, knowing it's squota and balance makes me feel better
that btrfs on top of raid5 isn't as unsafe as it used to be (it's
supposed to be safe, but I've had more than 5 unrecoverable FS crashes
after swraid5 misbehaved when I was really hoping for just a bit of data
loss or data corruption that scrub would find and then I could move on.
Also, I'm pretty sure I had -m dup all these years, so it's sad it
didn't help)
Marc
--
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
^ permalink raw reply [flat|nested] 43+ messages in thread
* Re: BTRFS discard crash: failed to run delayed ref for logical 15506102321152 num_bytes 16384 type 182 action 2 ref_mod 1: -2 6.11.2)
2026-04-11 3:35 BTRFS discard crash: failed to run delayed ref for logical 15506102321152 num_bytes 16384 type 182 action 2 ref_mod 1: -2 6.11.2) Marc MERLIN
` (3 preceding siblings ...)
2026-04-13 17:52 ` Simple quota unsafe? RIP: 0010:__btrfs_free_extent.isra.0+0xc41/0x1020 [btrfs] / do_free_extent_accounting:2999: errno=-2 No such entry Marc MERLIN
@ 2026-04-17 3:43 ` David Disseldorp
2026-04-17 5:19 ` Marc MERLIN
4 siblings, 1 reply; 43+ messages in thread
From: David Disseldorp @ 2026-04-17 3:43 UTC (permalink / raw)
To: Marc MERLIN; +Cc: linux-btrfs
Hi Marc,
Not a btrfs dev, but just wanted to make an observation...
On Fri, 10 Apr 2026 20:35:33 -0700, Marc MERLIN wrote:
> I had btfrs filesystem on top of raid5 with 5 spinning drives.
> I mistakenly enabled discard by mistake which caused a crash when the discard thread tried
> to run (no discard on those drives)
From the look of your block-dev path and loaded modules, the filesystem
is atop a bcache caching layer. This would make it far from a simple
dev topology; bcache would be expected to invalidate discarded cache
regions regardless of underlying HDD discard support.
Are you able to reproduce the bug if you remove the bcache layer?
^ permalink raw reply [flat|nested] 43+ messages in thread* Re: BTRFS discard crash: failed to run delayed ref for logical 15506102321152 num_bytes 16384 type 182 action 2 ref_mod 1: -2 6.11.2)
2026-04-17 3:43 ` BTRFS discard crash: failed to run delayed ref for logical 15506102321152 num_bytes 16384 type 182 action 2 ref_mod 1: -2 6.11.2) David Disseldorp
@ 2026-04-17 5:19 ` Marc MERLIN
0 siblings, 0 replies; 43+ messages in thread
From: Marc MERLIN @ 2026-04-17 5:19 UTC (permalink / raw)
To: David Disseldorp; +Cc: linux-btrfs
On Fri, Apr 17, 2026 at 01:43:29PM +1000, David Disseldorp wrote:
> From the look of your block-dev path and loaded modules, the filesystem
> is atop a bcache caching layer. This would make it far from a simple
> dev topology; bcache would be expected to invalidate discarded cache
> regions regardless of underlying HDD discard support.
> Are you able to reproduce the bug if you remove the bcache layer?
Fair question, this used to an array running with bcache.
The cache has gone away, so it's running in degraded mode without cache,
with just an extra layer that hopefully does nothing.
That said, I reproduced this twice on both my laptops that are running
without raid5 and directly on ssd without bcache
All that said, thank you for making sure we're not missing some other
reason for a bug
Marc
--
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Home page: http://marc.merlins.org/ | PGP 7F55D5F27AAF9D08
^ permalink raw reply [flat|nested] 43+ messages in thread