From: Marc MERLIN <marc@merlins.org>
To: Josef Bacik <josef@toxicpanda.com>
Cc: Roman Mamedov <rm@romanrm.net>,
dsterba@suse.cz, Martin Steigerwald <martin@lichtvoll.de>,
linux-btrfs@vger.kernel.org, kernel-team@fb.com
Subject: Re: btrfs filled up dm-thin and df%: shows 8.4TB of data used when I'm only using 10% of that.
Date: Fri, 21 Feb 2020 17:06:37 -0800 [thread overview]
Message-ID: <20200222010637.GB31491@merlins.org> (raw)
In-Reply-To: <20200222000142.GA31491@merlins.org>
On Fri, Feb 21, 2020 at 04:01:42PM -0800, Marc MERLIN wrote:
> [68099.512978] BTRFS info (device dm-13): the free space cache file (5151004819456) is invalid, skip it
> [68100.575692] BTRFS info (device dm-13): the free space cache file (5160668495872) is invalid, skip it
> [68100.689222] BTRFS info (device dm-13): the free space cache file (5161742237696) is invalid, skip it
>
> I knew that filling up a btrfs filesystem was bad, but filling it the
> normal way makes it slow down enough that you usually know and fix it.
> Filling it by having an underlying dm-thin deny writes, is much worse (I expected
> it wouldn't be pretty though, which is why I had a cronjob to catch this before it
> happened, but I missed it due to the df bug).
It took a while, but it finished eventually. Seems that unmount tries
to fix a lot of stuff, which took a long time and only stopped once LVM
returned an error, which forced readonly and finally allowed unmount to
succeed.
[68696.784521] BTRFS info (device dm-13): the free space cache file (9260214779904) is invalid, skip it
[68766.967084] BTRFS: error (device dm-13) in btrfs_commit_transaction:2279: errno=-5 IO failure (Error while writing out transaction)
[68767.005592] BTRFS info (device dm-13): forced readonly
[68767.022448] BTRFS warning (device dm-13): Skipping commit of aborted transaction.
[68767.045741] BTRFS: error (device dm-13) in cleanup_transaction:1830: errno=-5 IO failure
[68767.070945] BTRFS info (device dm-13): delayed_refs has NO entry
I guess I'm probably the few (or first?) person using btrfs with dm-thin with over subscription
and running out of space due to another bug.
Would it make sense to add some filesystem tests to see how btrfs behaves when it gets IO denied
errors by the underlying LVM? (which obviously doesn't happen on regular drives)?
mount read only is a better idea, I'll do that for now:
gargamel:/mnt/btrfs_pool2/backup/ubuntu# df -h .
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vgds2-ubuntu 14T 8.5T 5.6T 61% /mnt/btrfs_pool2/backup/ubuntu
df is showing a value consistent with btrfs fi df, but of course, not the correct value
since I'm only using a 10th of that data.
You asked for a check, it's running but may take a while:
gargamel:~# btrfs check /dev/mapper/vgds2-ubuntu
Checking filesystem on /dev/mapper/vgds2-ubuntu
UUID: 905c90db-8081-4071-9c79-57328b8ac0d5
checking extents
checking free space cache
checking fs roots
checking only csum items (without verifying data)
I'll paste the completion when it's done.
Marc
--
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Home page: http://marc.merlins.org/
next prev parent reply other threads:[~2020-02-22 1:06 UTC|newest]
Thread overview: 26+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-01-31 14:31 [PATCH] btrfs: do not zero f_bavail if we have available space Josef Bacik
2020-01-31 20:06 ` Martin Steigerwald
2020-02-01 1:00 ` Qu Wenruo
2020-02-02 17:52 ` David Sterba
[not found] ` <CAKhhfD7S=kcKLRURdNFZ8H4beS8=XjFvnOQXche7+SVOGFGC_w@mail.gmail.com>
2020-02-19 9:17 ` Martin Steigerwald
2020-02-19 13:43 ` Marc MERLIN
2020-02-19 14:31 ` David Sterba
2020-02-19 15:36 ` Marc MERLIN
2020-02-19 17:50 ` Roman Mamedov
2020-02-19 22:21 ` Martin Steigerwald
2020-02-20 21:46 ` Marc MERLIN
2020-02-21 5:38 ` Marc MERLIN
2020-02-21 5:45 ` Roman Mamedov
2020-02-21 23:07 ` btrfs filled up dm-thin and df%: shows 8.4TB of data used when I'm only using 10% of that Marc MERLIN
2020-02-21 23:17 ` How to roll back btrfs filesystem a few revisions? Marc MERLIN
2020-02-21 23:47 ` Josef Bacik
2020-02-22 0:08 ` Marc MERLIN
2020-02-22 0:36 ` Josef Bacik
2020-02-21 23:43 ` btrfs filled up dm-thin and df%: shows 8.4TB of data used when I'm only using 10% of that Josef Bacik
2020-02-22 0:01 ` Marc MERLIN
2020-02-22 0:43 ` Josef Bacik
2020-02-22 1:06 ` Marc MERLIN [this message]
2020-02-22 1:23 ` Marc MERLIN
2020-02-22 14:51 ` Marc MERLIN
2020-02-22 14:52 ` Josef Bacik
2020-02-22 15:24 ` Marc MERLIN
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20200222010637.GB31491@merlins.org \
--to=marc@merlins.org \
--cc=dsterba@suse.cz \
--cc=josef@toxicpanda.com \
--cc=kernel-team@fb.com \
--cc=linux-btrfs@vger.kernel.org \
--cc=martin@lichtvoll.de \
--cc=rm@romanrm.net \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox