Re: btrfs filled up dm-thin and df%: shows 8.4TB of data used when I'm only using 10% of that.

Marc MERLIN
2020-02-22T01:06:45Z

On Fri, Feb 21, 2020 at 04:01:42PM -0800, Marc MERLIN wrote:
> [68099.512978] BTRFS info (device dm-13): the free space cache file (5151004819456) is invalid, skip it
> [68100.575692] BTRFS info (device dm-13): the free space cache file (5160668495872) is invalid, skip it
> [68100.689222] BTRFS info (device dm-13): the free space cache file (5161742237696) is invalid, skip it
>
> I knew that filling up a btrfs filesystem was bad, but filling it the
> normal way makes it slow down enough that you usually know and fix it.
> Filling it by having an underlying dm-thin deny writes, is much worse (I expected
> it wouldn't be pretty though, which is why I had a cronjob to catch this before it
> happened, but I missed it due to the df bug).

It took a while, but it finished eventually. Seems that unmount tries
to fix a lot of stuff, which took a long time and only stopped once LVM
returned an error, which forced readonly and finally allowed unmount to
succeed.
[68696.784521] BTRFS info (device dm-13): the free space cache file (9260214779904) is invalid, skip it
[68766.967084] BTRFS: error (device dm-13) in btrfs_commit_transaction:2279: errno=-5 IO failure (Error while writing out transaction)
[68767.005592] BTRFS info (device dm-13): forced readonly
[68767.022448] BTRFS warning (device dm-13): Skipping commit of aborted transaction.
[68767.045741] BTRFS: error (device dm-13) in cleanup_transaction:1830: errno=-5 IO failure
[68767.070945] BTRFS info (device dm-13): delayed_refs has NO entry

I guess I'm probably the few (or first?) person using btrfs with dm-thin with over subscription
and running out of space due to another bug.
Would it make sense to add some filesystem tests to see how btrfs behaves when it gets IO denied
errors by the underlying LVM? (which obviously doesn't happen on regular drives)?

mount read only is a better idea, I'll do that for now:
gargamel:/mnt/btrfs_pool2/backup/ubuntu# df -h .
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vgds2-ubuntu 14T 8.5T 5.6T 61% /mnt/btrfs_pool2/backup/ubuntu

df is showing a value consistent with btrfs fi df, but of course, not the correct value
since I'm only using a 10th of that data.

You asked for a check, it's running but may take a while:
gargamel:~# btrfs check /dev/mapper/vgds2-ubuntu
Checking filesystem on /dev/mapper/vgds2-ubuntu
UUID: 905c90db-8081-4071-9c79-57328b8ac0d5
checking extents
checking free space cache
checking fs roots
checking only csum items (without verifying data)

I'll paste the completion when it's done.

Marc
--
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.

Home page: http://marc.merlins.org/