Re: btrfs raid1 and btrfs raid10 arrays NOT REDUNDANT

Chris Mason
2014-01-04T14:51:29Z

On Sat, 2014-01-04 at 06:10 +0000, Duncan wrote:
> Chris Murphy posted on Fri, 03 Jan 2014 16:22:44 -0700 as excerpted:
>
> > I would not make this option persistent by putting it permanently in the
> > grub.cfg; although I don't know the consequence of always mounting with
> > degraded even if not necessary it could have some negative effects (?)
>
> Degraded only actually does anything if it's actually needed. On a
> normal array it'll be a NOOP, so should be entirely safe for /normal/
> operation, but that doesn't mean I'd /recommend/ it for normal operation,
> since it bypasses checks that are there for a reason, thus silently
> bypassing information that an admin needs to know before he boots it
> anyway, in ordered to recover.
>

> However, I've some other comments to add:
>
> 1) As you I'm uncomfortable with the whole idea of adding degraded
> permanently at this point.
>

I added mount -o degraded just because I wanted the admin to be notified
of failures. Right now it's still the most reliable way to notify them,
but I definitely agree we can do better. Leaving it on all the time? I
don't think this is a great long term solution, unless you are actively
monitoring the system to make sure there are no failures.

Also, as Neil Brown pointed out it does put you at risk of transient
device detection failures getting things out of sync.

> Test:
>
> a) Create a two device btrfs raid1.
>
> b) Mount it and write some data to it.
>
> c) Unmount it, unplug one device, mount degraded the remaining device.
>
> d) Write some data to a test file on it, noting the path/filename and
> data.
>
> e) Unmount again, switch plugged devices so the formerly unplugged one is
> now the plugged one, and again mount degraded.
>
> f) Write some DIFFERENT data to the SAME path/file as in (d), so the two
> versions each on its own device have now incompatibly forked.
>
> g) Unmount, plug both devices in and mount, now undegraded.
>
> What I discovered back then, and to my knowledge the same behavior exists
> today, is that entirely unexpectedly from and in contrast to my mdraid
> experience, THE FILESYSTEM MOUNTED WITHOUT PROTEST!!
>
> h) I checked the file and one variant as written was returned. STILL NO
> WARNING! While I didn't test it, I'm assuming based on the PID-based
> round-robin read-assignment that I now know btrfs uses, that which copy I
> got would depend on whether the PID of the reading thread was even or
> odd, as that's what determines what device of the pair is read. (There
> has actually been some discussion of that as it's not a particularly
> intelligent balancing scheme and it's on the list to change, but the
> current even/odd works well enough for an initial implementation while
> the filesystem remains under development.)
>
> i) Were I rerunning the test today, I'd try a scrub and see what it did
> with the difference. But I was early enough in my btrfs learning that I
> didn't know to run it at that point, so didn't do so. I'd still be
> interested in how it handled that, tho based on what I know of btrfs
> behavior in general, I can /predict/ that which copy it'd scrub out and
> which it would keep, would again depend on the PID of the scrub thread,
> since both copies would appear valid (would verify against their checksum
> on the same device) when read, and it's only when matched against the
> other that a problem, presumably with the other copy, would be detected.
>

It'll pick the latest generation number and use that one as the one true
source. For the others you'll get crc errors which make it fall back to
the latest one. If the two have exactly the same generation number,
we'll have a hard time picking the best one.

Ilya has a series of changes from this year's GSOC that we need to clean
up and integrate. It detects offline devices and brings them up to date
automatically.

He targeted the pull-one-drive use case explicitly.

-chris