From: Marc MERLIN <marc@merlins.org>
To: Josef Bacik <josef@toxicpanda.com>
Cc: Roman Mamedov <rm@romanrm.net>,
dsterba@suse.cz, Martin Steigerwald <martin@lichtvoll.de>,
linux-btrfs@vger.kernel.org, kernel-team@fb.com
Subject: Re: btrfs filled up dm-thin and df%: shows 8.4TB of data used when I'm only using 10% of that.
Date: Fri, 21 Feb 2020 16:01:42 -0800 [thread overview]
Message-ID: <20200222000142.GA31491@merlins.org> (raw)
In-Reply-To: <3e94351d-6f32-1036-ab24-0dc1b843c969@toxicpanda.com>
On Fri, Feb 21, 2020 at 06:43:45PM -0500, Josef Bacik wrote:
> > Well, carap, see how 'used' went from 445.73GiB to 8.42TiB after balance?
>
> Wtf? Can you do btrfs filesystem usage on that fs? I'd like to see the
> breakdown. I'm super confused about what's happening there.
You and me both :)
gargamel:/mnt/btrfs_pool2/backup/ubuntu# btrfs fi df .
Data, single: total=8.40TiB, used=8.40TiB
System, DUP: total=8.00MiB, used=912.00KiB
Metadata, DUP: total=17.00GiB, used=16.33GiB
GlobalReserve, single: total=512.00MiB, used=0.00B
Looks like used is back to 8.4TB there too.
> > And now for extra points, this also damaged a 2nd of my filesystems on the same VG :(
> > [64723.601630] BTRFS error (device dm-17): bad tree block start, want 5782272294912 have 0
> > [64723.628708] BTRFS error (device dm-17): bad tree block start, want 5782272294912 have 0
> > [64897.028176] BTRFS error (device dm-13): parent transid verify failed on 22724608 wanted 10005 found 10001
> > [64897.080355] BTRFS error (device dm-13): parent transid verify failed on 22724608 wanted 10005 found 10001
>
> This will happen if the transaction aborts, does it still happen after you
> unmount and remount? Thanks,
the problematic filesystem mounts fine, but that doesn't mean it's
clean.
the one that I'd like very much not to be damaged, I'm not touching it
until I can get my VG back to having it's 50% of free space it needs to
have, with 99.9x%, it's not safe to use anything on it.
But thanks for the heads up that my other filesystem may be ok. I'll run
a btrfs check on it later when it's safe.
Back to dm-13, it's now hung on umount, I'm getting a string of these:
[67980.657803] BTRFS info (device dm-13): the free space cache file (4344624709632) is invalid, skip it
[67991.562812] BTRFS info (device dm-13): the free space cache file (4447703924736) is invalid, skip it
[67991.755262] BTRFS info (device dm-13): the free space cache file (4448777666560) is invalid, skip it
[68000.379059] BTRFS info (device dm-13): the free space cache file (4518570885120) is invalid, skip it
[68013.462077] BTRFS info (device dm-13): the free space cache file (4574405459968) is invalid, skip it
[68015.286730] BTRFS info (device dm-13): the free space cache file (4589437845504) is invalid, skip it
[68015.318239] BTRFS info (device dm-13): the free space cache file (4589437845504) is invalid, skip it
[68016.212246] BTRFS info (device dm-13): the free space cache file (4596954038272) is invalid, skip it
[68016.730826] BTRFS info (device dm-13): the free space cache file (4602322747392) is invalid, skip it
[68020.547135] BTRFS info (device dm-13): the free space cache file (4634535002112) is invalid, skip it
[68021.812820] BTRFS info (device dm-13): the free space cache file (4646346162176) is invalid, skip it
[68037.173441] BTRFS info (device dm-13): the free space cache file (4768752730112) is invalid, skip it
[68039.559383] BTRFS info (device dm-13): the free space cache file (4778416406528) is invalid, skip it
[68040.531083] BTRFS info (device dm-13): the free space cache file (4781637632000) is invalid, skip it
[68050.184300] BTRFS info (device dm-13): the free space cache file (4843914657792) is invalid, skip it
[68074.134080] BTRFS info (device dm-13): the free space cache file (4988869804032) is invalid, skip it
[68078.943126] BTRFS info (device dm-13): the free space cache file (5015713349632) is invalid, skip it
[68099.512978] BTRFS info (device dm-13): the free space cache file (5151004819456) is invalid, skip it
[68100.575692] BTRFS info (device dm-13): the free space cache file (5160668495872) is invalid, skip it
[68100.689222] BTRFS info (device dm-13): the free space cache file (5161742237696) is invalid, skip it
I knew that filling up a btrfs filesystem was bad, but filling it the
normal way makes it slow down enough that you usually know and fix it.
Filling it by having an underlying dm-thin deny writes, is much worse (I expected
it wouldn't be pretty though, which is why I had a cronjob to catch this before it
happened, but I missed it due to the df bug).
Marc
--
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Home page: http://marc.merlins.org/
next prev parent reply other threads:[~2020-02-22 0:01 UTC|newest]
Thread overview: 26+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-01-31 14:31 [PATCH] btrfs: do not zero f_bavail if we have available space Josef Bacik
2020-01-31 20:06 ` Martin Steigerwald
2020-02-01 1:00 ` Qu Wenruo
2020-02-02 17:52 ` David Sterba
[not found] ` <CAKhhfD7S=kcKLRURdNFZ8H4beS8=XjFvnOQXche7+SVOGFGC_w@mail.gmail.com>
2020-02-19 9:17 ` Martin Steigerwald
2020-02-19 13:43 ` Marc MERLIN
2020-02-19 14:31 ` David Sterba
2020-02-19 15:36 ` Marc MERLIN
2020-02-19 17:50 ` Roman Mamedov
2020-02-19 22:21 ` Martin Steigerwald
2020-02-20 21:46 ` Marc MERLIN
2020-02-21 5:38 ` Marc MERLIN
2020-02-21 5:45 ` Roman Mamedov
2020-02-21 23:07 ` btrfs filled up dm-thin and df%: shows 8.4TB of data used when I'm only using 10% of that Marc MERLIN
2020-02-21 23:17 ` How to roll back btrfs filesystem a few revisions? Marc MERLIN
2020-02-21 23:47 ` Josef Bacik
2020-02-22 0:08 ` Marc MERLIN
2020-02-22 0:36 ` Josef Bacik
2020-02-21 23:43 ` btrfs filled up dm-thin and df%: shows 8.4TB of data used when I'm only using 10% of that Josef Bacik
2020-02-22 0:01 ` Marc MERLIN [this message]
2020-02-22 0:43 ` Josef Bacik
2020-02-22 1:06 ` Marc MERLIN
2020-02-22 1:23 ` Marc MERLIN
2020-02-22 14:51 ` Marc MERLIN
2020-02-22 14:52 ` Josef Bacik
2020-02-22 15:24 ` Marc MERLIN
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20200222000142.GA31491@merlins.org \
--to=marc@merlins.org \
--cc=dsterba@suse.cz \
--cc=josef@toxicpanda.com \
--cc=kernel-team@fb.com \
--cc=linux-btrfs@vger.kernel.org \
--cc=martin@lichtvoll.de \
--cc=rm@romanrm.net \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox