BTRF: no free space

BTRF: no free space

[Reindl Harald]

this is a *new* filesystem and somewhere in the middle of the night at 
10% usage a large rsync stopped because BTRF thinks the filesystem is 
full - how is this possible in 2023?

Metadata, DUP: total=1.00GiB, used=945.61MiB

-----------------

defaults,noatime,compress-force=zstd:15,ssd,ssd_spread,nodiscard,nobarrier,noexec,nosuid,nodev,commit=60

adding "clear_cache" which solved such issues in the past eve refuses to 
mount it

[root@arrakisvm:~]$ mount /mnt/fileserver-backup/
mount: /mnt/fileserver-backup: mount(2) system call failed: No space 
left on device.

-----------------

[root@arrakisvm:~]$ df
Filesystem     Type   Size  Used Avail Use% Mounted on
/dev/sdd1      btrfs  2.0T  209G     0 100% /mnt/fileserver-backup

[root@arrakisvm:~]$ btrfs fi df /mnt/fileserver-backup
Data, single: total=1.95TiB, used=207.05GiB
System, DUP: total=8.00MiB, used=224.00KiB
Metadata, DUP: total=1.00GiB, used=945.61MiB
GlobalReserve, single: total=512.00MiB, used=0.00B

[root@arrakisvm:~]$ btrfs balance start -dusage=5 /mnt/fileserver-backup
ERROR: error during balancing '/mnt/fileserver-backup': No space left on 
device

[root@arrakisvm:~]$ dmesg
[Sun Oct 22 12:31:52 2023] BTRFS info (device sdd1): balance: start 
-dusage=5
[Sun Oct 22 12:31:54 2023] BTRFS info (device sdd1): 496 enospc errors 
during balance
[Sun Oct 22 12:31:54 2023] BTRFS info (device sdd1): balance: ended with 
status: -28

Re: BTRF: no free space

[Roman Mamedov]

On Sun, 22 Oct 2023 12:43:57 +0200
Reindl Harald <h.reindl@thelounge.net> wrote:

> this is a *new* filesystem and somewhere in the middle of the night at 
> 10% usage a large rsync stopped because BTRF thinks the filesystem is 
> full - how is this possible in 2023?
> 
> Metadata, DUP: total=1.00GiB, used=945.61MiB
> 
> -----------------
> 
> defaults,noatime,compress-force=zstd:15,ssd,ssd_spread,nodiscard,nobarrier,noexec,nosuid,nodev,commit=60
> 
> adding "clear_cache" which solved such issues in the past eve refuses to 
> mount it
> 
> [root@arrakisvm:~]$ mount /mnt/fileserver-backup/
> mount: /mnt/fileserver-backup: mount(2) system call failed: No space 
> left on device.
> 
> -----------------
> 
> [root@arrakisvm:~]$ df
> Filesystem     Type   Size  Used Avail Use% Mounted on
> /dev/sdd1      btrfs  2.0T  209G     0 100% /mnt/fileserver-backup
> 
> [root@arrakisvm:~]$ btrfs fi df /mnt/fileserver-backup
> Data, single: total=1.95TiB, used=207.05GiB
> System, DUP: total=8.00MiB, used=224.00KiB
> Metadata, DUP: total=1.00GiB, used=945.61MiB
> GlobalReserve, single: total=512.00MiB, used=0.00B

1) Report your kernel version

2) Retry without "ssd" and especially without "ssd_spread". The weird thing
here is that it appears the FS has allocated the entire unused space as "data"
chunks, leaving no space for any new metadata chunk. Personally I don't have
any other guess for the cause aside from the rarely-used (and little-tested?)
ssd_spread option. Of course still a bug even if so.

-- 
With respect,
Roman

Re: BTRF: no free space

[Reindl Harald]

Am 22.10.23 um 16:50 schrieb Roman Mamedov:
> On Sun, 22 Oct 2023 12:43:57 +0200
> Reindl Harald <h.reindl@thelounge.net> wrote:
> 
>> this is a *new* filesystem and somewhere in the middle of the night at
>> 10% usage a large rsync stopped because BTRF thinks the filesystem is
>> full - how is this possible in 2023?
>>
>> Metadata, DUP: total=1.00GiB, used=945.61MiB
>>
>> -----------------
>>
>> defaults,noatime,compress-force=zstd:15,ssd,ssd_spread,nodiscard,nobarrier,noexec,nosuid,nodev,commit=60
>>
>> adding "clear_cache" which solved such issues in the past eve refuses to
>> mount it
>>
>> [root@arrakisvm:~]$ mount /mnt/fileserver-backup/
>> mount: /mnt/fileserver-backup: mount(2) system call failed: No space
>> left on device.
>>
>> -----------------
>>
>> [root@arrakisvm:~]$ df
>> Filesystem     Type   Size  Used Avail Use% Mounted on
>> /dev/sdd1      btrfs  2.0T  209G     0 100% /mnt/fileserver-backup
>>
>> [root@arrakisvm:~]$ btrfs fi df /mnt/fileserver-backup
>> Data, single: total=1.95TiB, used=207.05GiB
>> System, DUP: total=8.00MiB, used=224.00KiB
>> Metadata, DUP: total=1.00GiB, used=945.61MiB
>> GlobalReserve, single: total=512.00MiB, used=0.00B
> 
> 1) Report your kernel version

6.5.7-100.fc37.x86_64

> 2) Retry without "ssd" and especially without "ssd_spread". The weird thing
> here is that it appears the FS has allocated the entire unused space as "data"
> chunks, leaving no space for any new metadata chunk. Personally I don't have
> any other guess for the cause aside from the rarely-used (and little-tested?)
> ssd_spread option. Of course still a bug even if so.

no - it allocated METADATA
Metadata, DUP: total=1.00GiB, used=945.61MiB

that was down from 100% after deleting a ton of data especially small files

"btrfs filesystem resize -100M" fixed the problem - it's pervert

https://superuser.com/questions/1310581/btrfs-on-ssd-no-space-left-on-device-catch-22-with-fstrim-and-btrfs-balan

Re: BTRF: no free space

[Roman Mamedov]

On Sun, 22 Oct 2023 17:40:05 +0200
Reindl Harald <h.reindl@thelounge.net> wrote:

> >> [root@arrakisvm:~]$ btrfs fi df /mnt/fileserver-backup
> >> Data, single: total=1.95TiB, used=207.05GiB
> >> System, DUP: total=8.00MiB, used=224.00KiB
> >> Metadata, DUP: total=1.00GiB, used=945.61MiB
> >> GlobalReserve, single: total=512.00MiB, used=0.00B
> > 
> > 1) Report your kernel version
> 
> 6.5.7-100.fc37.x86_64
> 
> > 2) Retry without "ssd" and especially without "ssd_spread". The weird thing
> > here is that it appears the FS has allocated the entire unused space as "data"
> > chunks, leaving no space for any new metadata chunk. Personally I don't have
> > any other guess for the cause aside from the rarely-used (and little-tested?)
> > ssd_spread option. Of course still a bug even if so.
> 
> no - it allocated METADATA
> Metadata, DUP: total=1.00GiB, used=945.61MiB

In your output I see that 1.95 TB was allocated for data chunks, even if only
207 GB of those are used for actual data. My guess was that this was
preventing the creation of one more metadata chunk when it was needed, which
broke everything.


-- 
With respect,
Roman

Re: BTRF: no free space

[Qu Wenruo]

On 2023/10/22 21:13, Reindl Harald wrote:
> this is a *new* filesystem and somewhere in the middle of the night at
> 10% usage a large rsync stopped because BTRF thinks the filesystem is
> full - how is this possible in 2023?
>
> Metadata, DUP: total=1.00GiB, used=945.61MiB

Metadata is already exhausted.

Thus no way to start balance.

>
> -----------------
>
> defaults,noatime,compress-force=zstd:15,ssd,ssd_spread,nodiscard,nobarrier,noexec,nosuid,nodev,commit=60
>
> adding "clear_cache" which solved such issues in the past eve refuses to
> mount it
>
> [root@arrakisvm:~]$ mount /mnt/fileserver-backup/
> mount: /mnt/fileserver-backup: mount(2) system call failed: No space
> left on device.
>
> -----------------
>
> [root@arrakisvm:~]$ df
> Filesystem     Type   Size  Used Avail Use% Mounted on
> /dev/sdd1      btrfs  2.0T  209G     0 100% /mnt/fileserver-backup
>
> [root@arrakisvm:~]$ btrfs fi df /mnt/fileserver-backup
> Data, single: total=1.95TiB, used=207.05GiB

Is this after you deleted some files? Or exactly where the situation is
when the ENOSPC happens?

If it's the latter case, it looks like a kernel bug that we're
over-allocating data chunks, and we need to investigate.
But please provide the initial 'btrfs fi df' output for that case.

If it's the former case and you're trying to make up enough space, I'm
afraid you have to delete more data until one continous 1G chunk is
freed up, then you can do balance to free up most of the space.

Thanks,
Qu

> System, DUP: total=8.00MiB, used=224.00KiB
> Metadata, DUP: total=1.00GiB, used=945.61MiB
> GlobalReserve, single: total=512.00MiB, used=0.00B
>
> [root@arrakisvm:~]$ btrfs balance start -dusage=5 /mnt/fileserver-backup
> ERROR: error during balancing '/mnt/fileserver-backup': No space left on
> device
>
> [root@arrakisvm:~]$ dmesg
> [Sun Oct 22 12:31:52 2023] BTRFS info (device sdd1): balance: start
> -dusage=5
> [Sun Oct 22 12:31:54 2023] BTRFS info (device sdd1): 496 enospc errors
> during balance
> [Sun Oct 22 12:31:54 2023] BTRFS info (device sdd1): balance: ended with
> status: -28
>
>

Re: BTRF: no free space

[Reindl Harald]

Am 22.10.23 um 23:33 schrieb Qu Wenruo:
> On 2023/10/22 21:13, Reindl Harald wrote:
>> this is a *new* filesystem and somewhere in the middle of the night at
>> 10% usage a large rsync stopped because BTRF thinks the filesystem is
>> full - how is this possible in 2023?
>>
>> Metadata, DUP: total=1.00GiB, used=945.61MiB
> 
> Metadata is already exhausted. 
> Thus no way to start balance.

pervert on a fresh 2 TB filesystem with 200 GB data in 2023 especially 
because i found "btrfs filesystem resize -100M" as solution which worked 
and afterwards balance was possible again

nearly as laughable as a small ext4 filesystem come swith defualts that 
you get a warning it's not useable after 2038

Re: BTRF: no free space

[Qu Wenruo]

On 2023/10/23 17:25, Reindl Harald wrote:
>
>
> Am 22.10.23 um 23:33 schrieb Qu Wenruo:
>> On 2023/10/22 21:13, Reindl Harald wrote:
>>> this is a *new* filesystem and somewhere in the middle of the night at
>>> 10% usage a large rsync stopped because BTRF thinks the filesystem is
>>> full - how is this possible in 2023?
>>>
>>> Metadata, DUP: total=1.00GiB, used=945.61MiB
>>
>> Metadata is already exhausted. Thus no way to start balance.
>
> pervert on a fresh 2 TB filesystem with 200 GB data in 2023 especially
> because i found "btrfs filesystem resize -100M" as solution which worked
> and afterwards balance was possible again

Then this is a little different.

If you can later reclaim space, it means your previous deletion indeed
emptied several data chunks, but they are not removed until cleaner get
kicked in.

Thanks,
Qu
>
> nearly as laughable as a small ext4 filesystem come swith defualts that
> you get a warning it's not useable after 2038