Re: How to handle a RAID5 arrawy with a failing drive? -> raid5 mostly works, just no rebuilds

Duncan
2014-03-20T07:37:31Z

Marc MERLIN posted on Wed, 19 Mar 2014 08:40:31 -0700 as excerpted:

> That's the thing though. If the bad device hadn't been forcibly removed,
> and apparently the only way to do this was to unmount, make the device
> node disappear, and remount in degraded mode, it looked to me like btrfs
> was still consideing that the drive was part of the array and trying to
> write to it.
> After adding a drive, I couldn't quite tell if it was striping over 11
> drive2 or 10, but it felt that at least at times, it was striping over
> 11 drives with write failures on the missing drive.
> I can't prove it, but I'm thinking the new data I was writing was being
> striped in degraded mode.

FWIW, there's at least two problems here, one a bug (or perhaps it'd more
accurately be described as an as yet incomplete feature) unrelated to
btrfs raid5/6 mode, the other the incomplete raid5/6 support. Both are
known issues, however.

The incomplete raid5/6 is discussed well enough elsewhere including in
this thread as a whole, which leaves the other issue.

The other issue, not specifically raid5/6 mode related, is that
currently, in-kernel btrfs is basically oblivious to disappearing drives,
thus explaining some of the more complex bits of the behavior you
described. Yes, the kernel has the device data and other layers know
when a device goes missing, but it's basically a case of the right hand
not knowing what the left hand is doing -- once setup on a set of
devices, in-kernel btrfs basically doesn't do anything with the device
information available to it, at least in terms of removing a device from
its listing when it goes missing. (It does seem to transparently handle
a missing btrfs component device reappearing, arguably /too/
transparently!)

Basically all btrfs does is log errors when a component device
disappears. It doesn't do anything with the disappeared device, and
really doesn't "know" it has disappeared at all, until an unmount and
(possibly degraded) remount, at which point it re-enumerates the devices
and again knows what's actually there... until a device disappears again.

There's actually patches being worked on to fix that situation as we
speak, and it's possible they're actually in btrfs-next already. (I've
seen the patches and discussion go by on the list but haven't tracked
them to the extent that I know current status, other than that they're
not in mainline yet.)

Meanwhile, counter-intuitively, btrfs-userspace is sometimes more aware
of current device status than btrfs-kernel is ATM, since parts of
userspace actually either get current status from the kernel, or trigger
a rescan in ordered to get it. But even after a rescan updates what
userspace knows and thus what the kernel as a whole knows, btrfs-kernel
still doesn't actually use that new information available to it in the
same kernel that btrfs-userspace used to get it from!

Knowing that rather counterintuitive "little" inconsistency, that isn't
actually so little, goes quite a way toward explaining what otherwise
looks like illogical btrfs behavior -- how could kernel-btrfs not know
the status of its own devices?

--
Duncan - List replies preferred. No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master." Richard Stallman