Re: Problem with file system

[Chris Murphy <lists@colorremedies.com>]

On Fri, Nov 3, 2017 at 10:46 PM, Adam Borowski <kilobyte@angband.pl> wrote:
> On Fri, Nov 03, 2017 at 04:03:44PM -0600, Chris Murphy wrote:
>> On Tue, Oct 31, 2017 at 5:28 AM, Austin S. Hemmelgarn
>> <ahferroin7@gmail.com> wrote:
>>
>> > If you're running on an SSD (or thinly provisioned storage, or something
>> > else which supports discards) and have the 'discard' mount option enabled,
>> > then there is no backup metadata tree (this issue was mentioned on the list
>> > a while ago, but nobody ever replied),
>>
>>
>> This is a really good point. I've been running discard mount option
>> for some time now without problems, in a laptop with Samsung
>> Electronics Co Ltd NVMe SSD Controller SM951/PM951.
>>
>> However, just trying btrfs-debug-tree -b on a specific block address
>> for any of the backup root trees listed in the super, only the current
>> one returns a valid result.  All others fail with checksum errors. And
>> even the good one fails with checksum errors within seconds as a new
>> tree is created, the super updated, and Btrfs considers the old root
>> tree disposable and subject to discard.
>>
>> So absolutely if I were to have a problem, probably no rollback for
>> me. This seems to totally obviate a fundamental part of Btrfs design.
>
> How is this an issue?  Discard is issued only once we're positive there's no
> reference to the freed blocks anywhere.  At that point, they're also open
> for reuse, thus they can be arbitrarily scribbled upon.

If it's not an issue, then no one should ever need those backup slots
in the super and we should just remove them.

But in fact, we know people end up situations where they're needed for
either automatic recovery at mount time or explicitly calling
--usebackuproot. And in some cases we're seeing users using discard
who have a borked root tree, and none of the backup roots are present
so they're fucked. Their file system is fucked.

Now again, maybe this means the hardware is misbehaving, and honored
the discard out of order, and did that and wrote the new supers before
it had completely committed all the metadata? I have no idea, but the
evidence is present in the list that some people run into this and
when they do the file system is beyond repair even t hough it can
usually be scraped with btrfs restore.


> Unless your hardware is seriously broken (such as lying about barriers,
> which is nearly-guaranteed data loss on btrfs anyway), there's no way the
> filesystem will ever reference such blocks.  The corpses of old trees that
> are left lying around with no discard can at most be used for manual
> forensics, but whether a given block will have been overwritten or not is
> a matter of pure luck.

File systems that overwrite, are hinting the intent in the journal
what's about to happen. So if there's a partial overwrite of metadata,
it's fine. The journal can help recover. But Btrfs without a journal,
has a major piece of information required to bootstrap the file system
at mount time, that's damaged, and then every backup has been
discarded. So it actually makes Btrfs more fragile than other file
systems in the same situation.



>
> For rollbacks, there are snapshots.  Once a transaction has been fully
> committed, the old version is considered gone.

Yeah well snapshots do not cause root trees to stick around.


>
>>  because it's already been discarded.
>> > This is ideally something which should be addressed (we need some sort of
>> > discard queue for handling in-line discards), but it's not easy to address.
>>
>> Discard data extents, don't discard metadata extents? Or put them on a
>> substantial delay.
>
> Why would you special-case metadata?  Metadata that points to overwritten or
> discarded blocks is of no use either.

I would rather lose 30 seconds, 1 minute, or even 2 minutes of writes,
than lose an entire file system. That's why.

Anyway right now I consider discard mount option fundamentally broken
on Btrfs for SSDs. I haven't tested this on LVM thinp, maybe it's
broken there too.

Even fstrim leaves a tiny window open for a few minutes every time it
gets called, where if the root tree is corrupted for any reason,
you're fucked because all the backup roots are already gone.

-- 
Chris Murphy