Re: Btrfs ENOSPC / Stuck in RO with "exclusive operation balance paused in progress"

[Alban Browaeys <alban.browaeys@gmail.com>]

Have you seen I posted a link to a btrfs debug image in a previous  email in this thread?

"
partition sdb3_var_metadata-only-debug.img at https://drive.google.com/file/d/1OzEZLaTUNpWsL1ohGKOMW4GiR9oQakdP/view?usp=drive_link done with "btrfs-image -c 9 -t 4 /dev/sdb3 /mnt/4/hermes-var-20260610-enospc-vanilla/sdb3_var_metadata-only-debug.img". 
"

I made an image and a btrfs debug image, is there any use for me to keep the broken on disk partition? I believe I cannot expect to be able to recover this on disk partition in a week and thus should delete it and recreate it if I want the pc back?
And keep the disk partition image for debugging.

Le 15/06/2026 à 22:00, Chris Murphy a écrit :
> 
> 
> On Mon, Jun 15, 2026, at 5:06 AM, Alban Browaeys wrote:
> 
>>  # btrfs check --readonly /dev/sdb3
>>  Opening filesystem to check...
>>  Checking filesystem on /dev/sdb3
>>  UUID: 13af326c-631f-482b-9c34-b59b4f100608
>>  [1/8] checking log skipped (none written)
>>  [2/8] checking root items
>>  [3/8] checking extents
>>  [4/8] checking free space tree
>>  [5/8] checking fs roots
>>  [6/8] checking only csums items (without verifying data)
>>  [7/8] checking root refs
>>  [8/8] checking quota groups skipped (not enabled on this FS)
>>  found 54982475776 bytes used, no error found
>>  total csum bytes: 50964248
>>  total tree bytes: 2142486528
>>  total fs tree bytes: 1951760384
>>  total extent tree bytes: 97910784
>>  btree space waste bytes: 569359506
>>  file data blocks allocated: 105614794752
>>   referenced 100030590976
> 
> OK so the file system is consistent. And there's no log tree.
> 
> For some reason skip_balance is not able to abort the balance/convert before the file system runs into the problem and goes read-only. Read-only prevents us from doing everything else.
> 
> Clearly the file system wants to allocate at least one metadata block group but cannot, there's no unallocated space on the device and no way to resize or add more space.
> 
> At the moment it appears to be stuck. So it's a bug somewhere.
> 
> 


> 
>>  Ok. Though I believe that if btrfs-restore find incomplete zstd frame
>>  it might be that rsync will not be able to copy everything from the ro
>>  half converted btrfs partition?
> 
> I don't know what the btrfs restore complaint about zstd frames is about. That makes me think there is corrupt data blocks, and the block can't be decompressed by zstd. 


Could it be half transfered data during the conversion ?

> You could try:
> 
> mount -o ro $dev1 /mnt
> btrfs scrub start -Brd /mnt


# mount -o ro /dev/sdb3 /var
[820430.153273] BTRFS: device label SSDHOME devid 1 transid 44655875 /dev/sdb3 (8:19) scanned by mount (13961)
[820430.175471] BTRFS info (device sdb3): first mount of filesystem 13af326c-631f-482b-9c34-b59b4f100608
[820430.185087] BTRFS info (device sdb3): using crc32c checksum algorithm
[820430.221645] BTRFS info (device sdb3): enabling ssd optimizations
[820430.228181] BTRFS info (device sdb3): turning on async discard
[820430.234575] BTRFS info (device sdb3): enabling free space tree
# btrfs scrub start -Brd /var
WARNING: cannot create scrub data file, mkdir /v[820447.453110] BTRFS info (device sdb3): scrub: started on devid 1
ar/lib failed: Read-only file system, status recording disabled
WARNING: failed to open the progress status socket at /var/lib/btrfs/scrub.progress.13af326c-631f-482b-9c34-b59b4f100608: No such file or directory, progress cannot be queried
Starting scrub on devid 1
[820598.534072] BTRFS info (device sdb3): scrub: finished on devid 1 with status: 0

Scrub device /dev/sdb3 (id 1) done
Scrub started:    Fri Jun 19 01:51:24 2026
Status:           finished
Duration:         0:02:31
Total to scrub:   51.21GiB
Rate:             347.25MiB/s
Error summary:    no errors found


# btrfs fi df /var
Data, single: total=65.44GiB, used=49.21GiB
System, single: total=32.00MiB, used=16.00KiB
Metadata, single: total=3.00GiB, used=2.00GiB
GlobalReserve, single: total=181.56MiB, used=0.00B

 
> That will do a read-only scrub on the read-only file system. It can't fix anything but it will show if there's corrupted files.
> 
> In any case, the file system does mount ro normally. You can reliably get important information out. The most portable backup/restore for podman/docker/mobi is tar - I know they have internal backup commands that use tar but I do not know what options are actually used and therefore how to make a tar backup of your containers rather than relying on rsync - it's not my area. But I would investigate a more generic way of getting important containers out of the broken file system so you can preserve them no matter the file system or graph driver you decide to use.


Alban