Re: With Linux 5.5: Filesystem full while still 90 GiB free

[Josef Bacik <josef@toxicpanda.com>]

On 1/30/20 3:18 PM, Chris Murphy wrote:
> On Thu, Jan 30, 2020 at 1:02 PM Martin Steigerwald <martin@lichtvoll.de> wrote:
>>
>> Chris Murphy - 30.01.20, 17:37:42 CET:
>>> On Thu, Jan 30, 2020 at 3:41 AM Martin Steigerwald
>> <martin@lichtvoll.de> wrote:
>>>> Chris Murphy - 29.01.20, 23:55:06 CET:
>>>>> On Wed, Jan 29, 2020 at 2:20 PM Martin Steigerwald
>>>>
>>>> <martin@lichtvoll.de> wrote:
>>>>>> So if its just a cosmetic issue then I can wait for the patch to
>>>>>> land in linux-stable. Or does it still need testing?
>>>>>
>>>>> I'm not seeing it in linux-next. A reasonable short term work
>>>>> around
>>>>> is mount option 'metadata_ratio=1' and that's what needs more
>>>>> testing, because it seems decently likely mortal users will need
>>>>> an easy work around until a fix gets backported to stable. And
>>>>> that's gonna be a while, me thinks.
>>>>>
>>>>> Is that mount option sufficient? Or does it take a filtered
>>>>> balance?
>>>>> What's the most minimal balance needed? I'm hoping -dlimit=1
>>>>
>>>> Does not make a difference. I did:
>>>>
>>>> - mount -o remount,metadata_ratio=1 /daten
>>>> - touch /daten/somefile
>>>> - dd if=/dev/zero of=/daten/someotherfile bs=1M count=500
>>>> - sync
>>>> - df still reporting zero space free
>>>>
>>>>> I can't figure out a way to trigger this though, otherwise I'd be
>>>>> doing more testing.
>>>>
>>>> Sure.
>>>>
>>>> I am doing the balance -dlimit=1 thing next. With metadata_ratio=0
>>>> again.
>>>>
>>>> % btrfs balance start -dlimit=1 /daten
>>>> Done, had to relocate 1 out of 312 chunks
>>>>
>>>> % LANG=en df -hT /daten
>>>> Filesystem             Type   Size  Used Avail Use% Mounted on
>>>> /dev/mapper/sata-daten btrfs  400G  311G     0 100% /daten
>>>>
>>>> Okay, doing with metadata_ratio=1:
>>>>
>>>> % mount -o remount,metadata_ratio=1 /daten
>>>>
>>>> % btrfs balance start -dlimit=1 /daten
>>>> Done, had to relocate 1 out of 312 chunks
>>>>
>>>> % LANG=en df -hT /daten
>>>> Filesystem             Type   Size  Used Avail Use% Mounted on
>>>> /dev/mapper/sata-daten btrfs  400G  311G     0 100% /daten
>>>>
>>>>
>>>> Okay, other suggestions? I'd like to avoid shuffling 311 GiB data
>>>> around using a full balance.
>>>
>>> There's earlier anecdotal evidence that -dlimit=10 will work. But you
>>> can just keep using -dlimit=1 and it'll balance a different block
>>> group each time (you can confirm/deny this with the block group
>>> address and extent count in dmesg for each balance). Count how many it
>>> takes to get df to stop misreporting. It may be a file system
>>> specific value.
>>
>> Lost the patience after 25 attempts:
>>
>> date; let I=I+1; echo "Balance $I"; btrfs balance start -dlimit=1 /daten
>> ; LANG=en df -hT /daten
>> Do 30. Jan 20:59:17 CET 2020
>> Balance 25
>> Done, had to relocate 1 out of 312 chunks
>> Filesystem             Type   Size  Used Avail Use% Mounted on
>> /dev/mapper/sata-daten btrfs  400G  311G     0 100% /daten
>>
>>
>> Doing the -dlimit=10 balance now:
>>
>> % btrfs balance start -dlimit=10 /daten ; LANG=en df -hT /daten
>> Done, had to relocate 10 out of 312 chunks
>> Filesystem             Type   Size  Used Avail Use% Mounted on
>> /dev/mapper/sata-daten btrfs  400G  311G     0 100% /daten
>>
>> Okay, enough of balancing for today.
>>
>> I bet I just wait for a proper fix, instead of needlessly shuffling data
>> around.
> 
> What about unmounting and remounting?
> 
> There is a proposed patch that David referenced in this thread, but
> it's looking like it papers over the real problem. But even if so,
> that'd get your file system working sooner than a proper fix, which I
> think (?) needs to be demonstrated to at least cause no new
> regressions in 5.6, before it'll be backported to stable.
> 
> 

The file system is fine, you don't need to balance or anything, this is purely a 
cosmetic bug.  _Always_ trust what btrfs filesystem usage tells you, and it's 
telling you that there's 88gib of unallocated space.  df is just wrong because 5 
years ago we arbitrarily decided to set b_avail to 0 if we didn't have enough 
metadata space for the whole global reserve, despite how much unallocated space 
we had left.  A recent changed means that we are more likely to not have enough 
free metadata space for the whole global reserve if there's unallocated space, 
specifically because we can use that unallocated space if we absolutely have to. 
  The fix will be to adjust the statfs() madness and then df will tell you the 
right thing (well as right as it can ever tell you anyway.)  Thanks,

Josef