Re: Please advise, strange "not enough to start the array while not clean"

[NeilBrown <neilb@suse.de>]

[-- Attachment #1: Type: text/plain, Size: 3292 bytes --]

On Mon, 22 Sep 2014 04:11:20 +0200 Patrik Horník <patrik@dsl.sk> wrote:

> Hello Neil,
> 
> I've got this situation unfamiliar to me on RAID6 array md1 with important data.
> 
> - It is RAID6 with 6 devices, 5 are partitions and 1 is another RAID0
> array md101 from two smaller drives. One of the smaller drives froze,
> so md101 got kicked out from md1 and marked as faulty in md1. After
> while I've stopped md1 without removing md101 from it first. Then I
> rebooted and assembled md101.
> 
> - First I tried mdadm -A --no-degraded -u UUID /dev/md1 but got
> "mdadm: /dev/md1 assembled from 5 drives (out of 6), but not started."
> so I stopped the md1.
> 
> - Second time I started it with -v and got:
> 
> mdadm: /dev/md101 is identified as a member of /dev/md1, slot 5.
> mdadm: /dev/sdk1 is identified as a member of /dev/md1, slot 4.
> mdadm: /dev/sdi1 is identified as a member of /dev/md1, slot 1.
> mdadm: /dev/sdh1 is identified as a member of /dev/md1, slot 2.
> mdadm: /dev/sdg1 is identified as a member of /dev/md1, slot 0.
> mdadm: /dev/sde1 is identified as a member of /dev/md1, slot 3.
> mdadm: added /dev/sdi1 to /dev/md1 as 1
> mdadm: added /dev/sdh1 to /dev/md1 as 2
> mdadm: added /dev/sde1 to /dev/md1 as 3
> mdadm: added /dev/sdk1 to /dev/md1 as 4
> mdadm: added /dev/md101 to /dev/md1 as 5 (possibly out of date)
> mdadm: added /dev/sdg1 to /dev/md1 as 0
> mdadm: /dev/md1 assembled from 5 drives (out of 6), but not started.
> 
> - On third time I tried without --nodegraded with mdadm -A -v -u UUID
> /dev/md1. This is what I've got:
> 
> mdadm: /dev/md101 is identified as a member of /dev/md1, slot 5.
> mdadm: /dev/sdk1 is identified as a member of /dev/md1, slot 4.
> mdadm: /dev/sdi1 is identified as a member of /dev/md1, slot 1.
> mdadm: /dev/sdh1 is identified as a member of /dev/md1, slot 2.
> mdadm: /dev/sdg1 is identified as a member of /dev/md1, slot 0.
> mdadm: /dev/sde1 is identified as a member of /dev/md1, slot 3.
> mdadm: added /dev/sdi1 to /dev/md1 as 1
> mdadm: added /dev/sdh1 to /dev/md1 as 2
> mdadm: added /dev/sde1 to /dev/md1 as 3
> mdadm: added /dev/sdk1 to /dev/md1 as 4
> mdadm: added /dev/md101 to /dev/md1 as 5 (possibly out of date)
> mdadm: added /dev/sdg1 to /dev/md1 as 0
> mdadm: /dev/md1 assembled from 5 drives - not enough to start the
> array while not clean - consider --force.
> 
> Array md1 has bitmap. All drive devices have all same Events, their
> state is clean and Device Role is Active device. md101 has active
> state and lower Events.
> 
> Is this expected behavior? My theory is that it is caused by md101 and
> I should start array md1 without it (by for example stopping md101)
> and then re-add it. Is that a case or is it something else?
> 
> Thanks.
> 
> Best regards,
> 
> Patrik


The array is clearly degraded as one of the devices failed and hasn't been
recovered yet, so using --nodegraded is counter productive, as you
discovered.

It appears that the array is also marked as 'dirty'.  That suggests that it
wasn't shut down cleanly.
What does "mdadm --examine" of some device show?

You probably need to re-assemble the array with --force like it suggests,
then add the failed device and let it recover.

NeilBrown


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]