Re: Out of space loop: skip_balance not working

[Qu Wenruo <quwenruo.btrfs@gmx.com>]

On 2023/6/13 09:54, Stefan N wrote:
> Hi Paul,
>
> Thanks for the suggestion, I've had similar success in the past, but
> unfortunately not this time.
>
> I'm guessing this is because the metadata is full rather than the data.
>
> If I do `mount && btrfs dev add` as a single command on a small loop
> device it still doesn't perform in time before it turns to read only.

This is because btrfs_init_new_device() would commit transaction.

In your particular case, since you're running RAID1C4 you need to add 4
devices in one transaction.

I can easily craft a patch to avoid commit transaction, but still you'll
need to add at least 4 disks, and then sync to see if things would work.

Furthermore this means you need a liveCD with full kernel compiling
environment.

If you want to go this path, I can send you the patch when you've
prepared the needed environment.

Thanks,
Qu

> The same goes if I try to rm, truncate, btrfs fi sync or btrfs
> balance.
>
> - Stefan
>
> On Tue, 13 Jun 2023 at 10:59, Paul Jones <paul@pauljones.id.au> wrote:
>>
>>
>>> -----Original Message-----
>>> From: Stefan N <stefannnau@gmail.com>
>>> Sent: Monday, June 12, 2023 11:03 PM
>>> To: Qu Wenruo <quwenruo.btrfs@gmx.com>
>>> Cc: linux-btrfs@vger.kernel.org
>>> Subject: Re: Out of space loop: skip_balance not working
>>>
>>> On Mon, 12 Jun 2023 at 20:16, Qu Wenruo <quwenruo.btrfs@gmx.com>
>>> wrote:
>>>>> I've tried using the latest ubuntu livecd which has btrfs-progs v6.2
>>>>> on kernel 6.20.0-20
>>>>
>>>> I guess you mean 6.2?
>>>
>>> Sorry yes kernel 6.2.0-20 (Ubuntu)
>>>
>>>> In v6.2 kernel Josef introduced a new mechanism called
>>> FLUSH_EMERGENCY
>>>> to try our best to squish out any extra metadata space.
>>>>
>>>> If that doesn't work, I'm running out of ideas.
>>>
>>> How do I go about forcing this to engage? Currently the array never stays in
>>> write mode long enough to do anything, so I'd need to trigger something
>>> immediately after mount to have a chance that it syncs before it goes into
>>> read only mode.
>>>
>>>>> BTRFS info (device sdi: state A): space_info total=83530612736,
>>>>> used=82789154816, pinned=245710848, reserved=495747072,
>>>>> may_use=130809856, readonly=0 zone_unusable=0
>>>>
>>>> The big concern here is, we have hundreds of MiBs for
>>>> pinned/reserved/may_use.
>>>>
>>>> Which looks too large.
>>>>
>>>> My concern is either free space tree or extent tree updates are taking
>>>> too much space.
>>>>
>>>> Have you tried to cancel the balance and sync? That doesn't work either?
>>>
>>> The balance cancels ok, and there's no sync running except the auto UUID
>>> tree check on mount.
>>>
>>>> Considering you have quite some data space left, you may want to
>>>> migrate to space cache v1.
>>>> Unlike v2 cache, v1 cache only takes data space, thus may squish out
>>>> some precious metadata space.
>>>
>>>  From the 'disk space caching is enabled' in the log it must still be using space
>>> cache v1, and forcing it as a flag doesn't appear to change anything.
>>>
>>> With many remount cycles, the best I've been able to achieve has been to rm
>>> some small files, but they never synced and were back in btrfs on remount.
>>>
>>> I'm running out of ideas, and given the size I really don't want to have to
>>> replace/rebuild if I can help it!
>>>
>>> Any other ideas would be greatly appreciated
>>
>>
>> When I've had similar issues in the past I've managed to create some space by adding a usb drive (or two) to the filesystem, which then gives enough of a buffer to remove some files, and when btrfs will let you remove the extra drive you know everything is back under control.
>>
>> Paul.