Re: Unexpected raid1 behaviour

["Austin S. Hemmelgarn" <ahferroin7@gmail.com>]

On 2017-12-17 10:48, Peter Grandi wrote:
> "Duncan"'s reply is slightly optimistic in parts, so some
> further information...
> 
> [ ... ]
> 
>> Basically, at this point btrfs doesn't have "dynamic" device
>> handling.  That is, if a device disappears, it doesn't know
>> it.
> 
> That's just the consequence of what is a completely broken
> conceptual model: the current way most multi-device profiles are
> designed is that block-devices and only be "added" or "removed",
> and cannot be "broken"/"missing". Therefore if IO fails, that is
> just one IO failing, not the entire block-device going away.
> The time when a block-device is noticed as sort-of missing is
> when it is not available for "add"-ing at start.
> 
> Put another way, the multi-device design is/was based on the
> demented idea that block-devices that are missing are/should be
> "remove"d, so that a 2-device volume with a 'raid1' profile
> becomes a 1-device volume with a 'single'/'dup' profile, and not
> a 2-device volume with a missing block-device and an incomplete
> 'raid1' profile, even if things have been awkwardly moving in
> that direction in recent years.
> 
> Note the above is not totally accurate today because various
> hacks have been introduced to work around the various issues.
You do realize you just restated exactly what Duncan said, just in a 
much more verbose (and aggressively negative) manner...
> 
>> Thus, if a device disappears, to get it back you really have
>> to reboot, or at least unload/reload the btrfs kernel module,
>> in ordered to clear the stale device state and have btrfs
>> rescan and reassociate devices with the matching filesystems.
> 
> IIRC that is not quite accurate: a "missing" device can be
> nowadays "replace"d (by "devid") or "remove"d, the latter
> possibly implying profile changes:
> 
>    https://btrfs.wiki.kernel.org/index.php/Using_Btrfs_with_Multiple_Devices#Using_add_and_delete
> 
> Terrible tricks like this also work:
> 
>    https://www.spinics.net/lists/linux-btrfs/msg48394.html
While that is all true, none of that _fixes_ the issue of a device 
disappearing and then being reconnected.  In theory, you can use `btrfs 
device replace` to force BTRFS to acknowledge the new name (by 
'replacing' the missing device with the now returned device), but doing 
so is horribly inefficient as to not be worth it unless you have no 
other choice.
> 
>> Meanwhile, as mentioned above, there's active work on proper
>> dynamic btrfs device tracking and management. It may or may
>> not be ready for 4.16, but once it goes in, btrfs should
>> properly detect a device going away and react accordingly,
> 
> I haven't seen that, but I doubt that it is the radical redesign
> of the multi-device layer of Btrfs that is needed to give it
> operational semantics similar to those of MD RAID, and that I
> have vaguely described previously.
Anand has been working on hot spare support, and as part of that has 
done some work on handling of missing devices.
> 
>> and it should detect a device coming back as a different
>> device too.
> 
> That is disagreeable because of poor terminology: I guess that
> what was intended that it should be able to detect a previous
> member block-device becoming available again as a different
> device inode, which currently is very dangerous in some vital
> situations.
How exactly is this dangerous?  The only situation I can think of is if 
a bogus device is hot-plugged and happens to perfectly match all the 
required identifiers, and at that point you've either got someone 
attacking your system who already has sufficient access to do whatever 
the hell they want with it, or you did something exceedingly stupid, and 
both cases are dangerous by themselves.
> 
>> Longer term, there's further patches that will provide a
>> hot-spare functionality, automatically bringing in a device
>> pre-configured as a hot- spare if a device disappears, but
>> that of course requires that btrfs properly recognize devices
>> disappearing and coming back first, so one thing at a time.
> 
> That would be trivial if the complete redesign of block-device
> states of the Btrfs multi-device layer happened, adding an
> "active" flag to an "accessible" flag to describe new member
> states, for example.
No, it wouldn't be trivial, because a complete redesign of part of the 
filesystem would be needed.
> 
> My guess that while logically consistent, the current
> multi-device logic is fundamentally broken from an operational
> point of view, and needs a complete replacement instead of
> fixes.
Then why don't you go write up some patches yourself if you feel so 
strongly about it?

The fact is, the only cases where this is really an issue is if you've 
either got intermittently bad hardware, or are dealing with external 
storage devices.  For the majority of people who are using multi-device 
setups, the common case is internally connected fixed storage devices 
with properly working hardware, and for that use case, it works 
perfectly fine.  In fact, the only people I've seen any reports of 
issues from are either:

1. Testing the behavior of device management (such as the OP), in which 
case, yes it doesn't work if you do things that aren't reasonably 
expected of working hardware.
2. Trying to do multi-device on USB, which is a bad idea regardless of 
what you're using to create a single volume, because USB has pretty 
serious reliability issues.

Neither case is 'normal' usage of a multi-device volume though.  Yes, 
the second case could be better supported, but that's likely going to 
require some help from the block layer, and verification of writes.  As 
far as handling of other marginal hardware, I'm very inclined to say 
that BTRFS should not care.  At the point at which a device is dropping 
off the bus and reappearing with enough regularity for this to be an 
issue, you have absolutely no idea how else it's corrupting your data, 
and support of such a situation is beyond any filesystem (including ZFS).