"failed" vs "removed" or "locked-out" state and --incremental auto-re-adding

"failed" vs "removed" or "locked-out" state and --incremental auto-re-adding

[Christian Gatzemeier]

Hi all,

I hope you can help in sorting out the different states unused
devices can be in, and the commands used to change states.

It seems we don't have an obvious way to manually
remove a member from an array so that it does not get auto-assembled
later on (i.e. by udev/--incremental) without completely zeroing the
superblock or moving it into an own array. Maybe a --lock-out
option would make sense here? (Recognizable maybe by a superblock marking itself
as "removed"?)
Locking-out members would be handy to keep snapshots without the need to unplug
the drives to make sure they don't get re-added. And we would need such a
lock-out possibility if someone would like to implement automatically locking
out segments with conflicting changes/alternative versions (after incidences
when they are both connected).


Another thing is that trying to --remove without prior --failing gives a rather
unintuitive "device busy":
Is there a reason that --remove (and --lock-out)
shouldn't just automatically "fail" a device?
Or the other way around when would one want to
manually --fail a member without subsequently --removing it?

Kind Regards,
Christian

Re: "failed" vs "removed" or "locked-out" state and --incremental auto-re-adding

[Phillip Susi]

On 4/23/2010 8:20 AM, Christian Gatzemeier wrote:
> It seems we don't have an obvious way to manually
> remove a member from an array so that it does not get auto-assembled
> later on (i.e. by udev/--incremental) without completely zeroing the
> superblock or moving it into an own array. Maybe a --lock-out

mdadm --fail followed by mdadm --remove seems to be what you are looking
for.  mdadm --incremental will not use a removed drive.

Actually though, it looks like mdadm does not update the metadata on the
disk when you --fail and --remove it, so it still thinks it is part of
an array.  This appears to be a bug.

Even though the second disk says it is still part of the array, mdadm
--incremental tries to add it to the array, the state of the first disk
takes over and the second disk remains removed.

> Another thing is that trying to --remove without prior --failing gives a rather
> unintuitive "device busy":
> Is there a reason that --remove (and --lock-out)
> shouldn't just automatically "fail" a device?
> Or the other way around when would one want to
> manually --fail a member without subsequently --removing it?

Might be a nice feature though the man page is rather clear that you
have to fail first, then remove.  You might want to fail it and not
remove it so that you can remove the drive for now, and plug it back in
later.

Re: "failed" vs "removed" or "locked-out" state and --incremental auto-re-adding

[Christian Gatzemeier]

As comming to "terms" working with mdadm took me a while, I'll add my current
"translations" of the actions to the discussion:

1) The "failed" state is the state a member that failed or is missing gets,
while it can stay listed in mdstat.

2) To "unbind", "unlist" or "dismiss" a member from the md device stats is
currently called to --remove it. In particular you can "unbind", "unlist" or
"dismiss" failed or detatched members with --remove failed/detached.

3) A safe way to "lock-out" or "really remove" members from udev/--incremental
assembly is not available yet AFAIK. (--zero-superblock on mirror members makes
the md device content detectable/available directly)


IMHO the ones mentioned first could seen as implied by those mentioned later.

I am unclear why --incremental seems to require a device to be unbound first
(--removed) in order to re-add it after it failed. IMHO it could do it itself if
it is really necessary without bothering the user.

Re: "failed" vs "removed" or "locked-out" state and --incremental auto-re-adding

[Doug Ledford]

[-- Attachment #1: Type: text/plain, Size: 3282 bytes --]

On 04/26/2010 06:28 PM, Christian Gatzemeier wrote:
> 
> As comming to "terms" working with mdadm took me a while, I'll add my current
> "translations" of the actions to the discussion:
> 
> 1) The "failed" state is the state a member that failed or is missing gets,
> while it can stay listed in mdstat.

Yes.  In particular failed simply means that the kernel no longer
considers it a running member of the array.  However, the kernel still
holds open the reference to the device (which means anything/anyone else
is still locked out from attempting to access the device, which prevents
anything bad from happening to the data that was on it when it failed).

> 2) To "unbind", "unlist" or "dismiss" a member from the md device stats is
> currently called to --remove it. In particular you can "unbind", "unlist" or
> "dismiss" failed or detatched members with --remove failed/detached.

You can use --remove failed/detached/<devname>, they all work.  But yes,
the underlying action here is to take an already failed device go ahead
and release all references to the device from the raid stack.  In
particular, this releases the exclusive open the raid stack holds on the
device and now makes the device available for other things to
open/modify.  At this point there is no longer any guarantee that the
device will not be modified from the pristine state it was in when it
failed.

> 3) A safe way to "lock-out" or "really remove" members from udev/--incremental
> assembly is not available yet AFAIK. (--zero-superblock on mirror members makes
> the md device content detectable/available directly)

This is a shortcoming of version 0.90/1.0 superblocks and raid1 arrays.
 For all other superblock versions and raid types, this is not true.
The default superblock version changed from 0.90 to 1.2 as of the mdadm
3.1 series and so this won't be a problem in the future.

> IMHO the ones mentioned first could seen as implied by those mentioned later.

No, and this is a safety feature.  We won't remove a good device in
order to prevent a typo from rendering an array dead.  Imagine that
/dev/sdd1 was already failed, and you typed mdadm /dev/md0 -r /dev/sdc1
and we just blindly failed and then removed sdc1, and assume the array
could only handle one failed member (aka, raid4 or raid5), you've just
rendered the array dead in the water.  We could ask questions I suppose,
but it's just as well off to require that a drive be failed before we
remove it.

> I am unclear why --incremental seems to require a device to be unbound first
> (--removed) in order to re-add it after it failed. IMHO it could do it itself if
> it is really necessary without bothering the user.

It would be kind of useless to put that support into incremental.
Incremental isn't really intended to be run from the command line
(although you can), it's intended to be done on hotplug events.  Those
hotplug events never happen when the device is failed but not removed
from an array, so it's a condition we don't need to handle.

-- 
Doug Ledford <dledford@redhat.com>
              GPG KeyID: CFBFF194
	      http://people.redhat.com/dledford

Infiniband specific RPMs available at
	      http://people.redhat.com/dledford/Infiniband


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

"failed" vs "released" and "locked-out" state and --incremental auto-re-adding

[Christian Gatzemeier]

Thanks you for responding and adding insight.

Doug Ledford <dledford <at> redhat.com> writes:
> On 04/26/2010 06:28 PM, Christian Gatzemeier wrote:
> > 2) To "unbind", "unlist" or "dismiss" a member from the md device stats is
> > currently called to --remove it. In particular you can "unbind", "unlist"
> > or "dismiss" failed or detatched members with --remove failed/detached.
> 
> You can use --remove failed/detached/≤devname>, they all work.  But yes,
> the underlying action here is to take an already failed device go ahead
> and release

There we have a very good word to name --remove so that mdadm is easier to
understand (IMHO). "release"



> > 3) A safe way to "lock-out" or "really remove" members from
> > udev/--incremental assembly is not available yet AFAIK.
> > (--zero-superblock on mirror members makes the md device content
> >  detectable/available directly)
> 
> This is a shortcoming of version 0.90/1.0 superblocks and raid1 arrays.
>  For all other superblock versions and raid types, this is not true.
> The default superblock version changed from 0.90 to 1.2 as of the mdadm
> 3.1 series and so this won't be a problem in the future.

Good, that'll fix the problem with --zero-superblock for the future.

An explicit "locked-out" state without killing the superblock may still be
good. (Even if only for pausing or optionally auto-locking out detected
alternative versions.)



 
> > IMHO the ones mentioned first could seen as implied by those mentioned
> > later.
> 
> No, and this is a safety feature.  We won't remove a good device in
> order to prevent a typo from rendering an array dead.

I understand, makes sense to me.
Ok, if mdadm --remove (release) could give a little hint to --fail first, if
"device is busy", it may be able save some head scratches. ;)



> > I am unclear why --incremental seems to require a device to be
> >  --removed (released) first
>
> It would be kind of useless to put that support into incremental.
> Incremental isn't really intended to be run from the command line
> (although you can), it's intended to be done on hotplug events.

That is exactly were I encountered this. Unplugging a failed disk, and
plugging it back in again would fail, unless I manually --remove (released)
the device before plugging it back in.

But I think the hot-unplugging support you added will probably fix this in
the future even nicer. (Automatically releasing devices as soon as they are
detached.)



--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: "failed" vs "released" and "locked-out" state and --incremental auto-re-adding

[Doug Ledford]

[-- Attachment #1: Type: text/plain, Size: 2289 bytes --]

On 04/27/2010 06:13 AM, Christian Gatzemeier wrote:
> 
> Thanks you for responding and adding insight.
> 
> Doug Ledford <dledford <at> redhat.com> writes:
>> On 04/26/2010 06:28 PM, Christian Gatzemeier wrote:
>>> 2) To "unbind", "unlist" or "dismiss" a member from the md device stats is
>>> currently called to --remove it. In particular you can "unbind", "unlist"
>>> or "dismiss" failed or detatched members with --remove failed/detached.
>>
>> You can use --remove failed/detached/≤devname>, they all work.  But yes,
>> the underlying action here is to take an already failed device go ahead
>> and release
> 
> There we have a very good word to name --remove so that mdadm is easier to
> understand (IMHO). "release"

You're probably right, but it's also too late to change it now :-(
Remove has been in use for quite some time and there are untold numbers
of programs and scripts that use it as it is so that it would be very
difficult to change it.

>> No, and this is a safety feature.  We won't remove a good device in
>> order to prevent a typo from rendering an array dead.
> 
> I understand, makes sense to me.
> Ok, if mdadm --remove (release) could give a little hint to --fail first, if
> "device is busy", it may be able save some head scratches. ;)

Very valid request.

>>> I am unclear why --incremental seems to require a device to be
>>>  --removed (released) first
>>
>> It would be kind of useless to put that support into incremental.
>> Incremental isn't really intended to be run from the command line
>> (although you can), it's intended to be done on hotplug events.
> 
> That is exactly were I encountered this. Unplugging a failed disk, and
> plugging it back in again would fail, unless I manually --remove (released)
> the device before plugging it back in.
> 
> But I think the hot-unplugging support you added will probably fix this in
> the future even nicer. (Automatically releasing devices as soon as they are
> detached.)

Yep, the hot unplug support solves this issue quite nicely.

-- 
Doug Ledford <dledford@redhat.com>
              GPG KeyID: CFBFF194
	      http://people.redhat.com/dledford

Infiniband specific RPMs available at
	      http://people.redhat.com/dledford/Infiniband


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: "failed" vs "released" and "locked-out" state and --incremental auto-re-adding

[Christian Gatzemeier]

Doug Ledford <dledford <at> redhat.com> writes:

> 
> On 04/27/2010 06:13 AM, Christian Gatzemeier wrote:
>
> > There we have a very good word to name --remove so that mdadm is easier to
> > understand (IMHO). "release"
> 
> You're probably right, but it's also too late to change it now 
> Remove has been in use for quite some time and there are untold numbers
> of programs and scripts that use it as it is so that it would be very
> difficult to change it.

I agree. The only thing one could probably do is adding a --release option that
just does the same as --remove, and maybe depreciating --remove when refering to
that action (but of course never actually removing it).

In any case thank you for your enhancements and fine explanations.


Christian