mdadm RAID1 with 3 disks active on 2 disks configuration

mdadm RAID1 with 3 disks active on 2 disks configuration

[Aymeric]

Hello,

I've a system with a RAID1 using two disks (sda + sdb) + 1 spare (sdc).

For some stupid reason (cable) the spare (sdc) became active and sda was
out the array.

I then added back sda to the RAID1 but now I've RAID1 with three active
disks and zero spare.
I don't understand why the spare device didn't switch back as spare
after sda have been rebuild.

I used the following command to add sda to the RAID1:
# mdadm  /dev/md0 --add /dev/sda3


What did I do wrong here? And how to fix it? Do I have to set sdc to
faulty and add it back?

Thanks!

Here are the RAID1 status:

# cat /proc/mdstat
Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5]
[raid4] [raid10]
md0 : active raid1 sda3[2] sdc3[3](W)(R) sdb3[0]
      976015360 blocks super 1.2 [2/3] [UU]
      bitmap: 1/8 pages [4KB], 65536KB chunk

# mdadm --detail /dev/md0
/dev/md0:
           Version : 1.2
     Creation Time : Wed Sep  5 09:45:55 2018
        Raid Level : raid1
        Array Size : 976015360 (930.80 GiB 999.44 GB)
     Used Dev Size : 976015360 (930.80 GiB 999.44 GB)
      Raid Devices : 2
     Total Devices : 3
       Persistence : Superblock is persistent

     Intent Bitmap : Internal

       Update Time : Sun Feb 23 18:28:45 2020
             State : clean
    Active Devices : 3
   Working Devices : 3
    Failed Devices : 0
     Spare Devices : 0

Consistency Policy : bitmap

              Name : sharon:2
              UUID : 98299cb7:9cbe61e9:c8374bcb:3c72d5ca
            Events : 704774

    Number   Major   Minor   RaidDevice State
       0       8       19        0      active sync   /dev/sdb3
       2       8        3        1      active sync   /dev/sda3
       3       8       35        1      active sync writemostly   /dev/sdc3


-- 
Aymeric

Re: mdadm RAID1 with 3 disks active on 2 disks configuration

[antlists]

On 23/02/2020 17:38, Aymeric wrote:
> Hello,
>
> I've a system with a RAID1 using two disks (sda + sdb) + 1 spare (sdc).
>
> For some stupid reason (cable) the spare (sdc) became active and sda was
> out the array.
>
> I then added back sda to the RAID1 but now I've RAID1 with three active
> disks and zero spare.
> I don't understand why the spare device didn't switch back as spare
> after sda have been rebuild.
Because rebuilding sda is a lot of work. The sensible option would have 
been to leave sdb and sdc active, and left sda as spare.
> I used the following command to add sda to the RAID1:
> # mdadm  /dev/md0 --add /dev/sda3
>
>
> What did I do wrong here? And how to fix it? Do I have to set sdc to
> faulty and add it back?

I'd just leave it as a 3-disk raid. But no, if you do want to set it 
back to a 2-disk raid don't do a "faulty and re-add" - that'll just make 
re-sync it again. Somehow you've got active devices set to 3, so just 
set that back to two. If you look on the wiki it covers the scenario of 
converting a three-disk raid to two.

The only thing I will add about a 3-disk raid is that if you want to 
change it to anything else, you'll have to revert to 2 first, but do 
that when you want to change things, leave it at 3 for the moment.

> Thanks!

Cheers,

Wol

Re: mdadm RAID1 with 3 disks active on 2 disks configuration

[Aymeric P.]

Le 23 février 2020 21:40:44 GMT+01:00, antlists <antlists@youngman.org.uk> a écrit :
>On 23/02/2020 17:38, Aymeric wrote:
>
>Because rebuilding sda is a lot of work. The sensible option would have
>
>been to leave sdb and sdc active, and left sda as spare.

Well, for some reason that didn't happen that way..

>> I used the following command to add sda to the RAID1:
>> # mdadm  /dev/md0 --add /dev/sda3
>>
>>
>> What did I do wrong here? And how to fix it? Do I have to set sdc to
>> faulty and add it back?
>
>I'd just leave it as a 3-disk raid. But no, if you do want to set it 
>back to a 2-disk raid don't do a "faulty and re-add" - that'll just
>make 
>re-sync it again.

I would prefer to revert to sdc as a spare, it's a really a slow disk (5400rpm with a sata1 while others are 7200 at sata3). For a spare disk it's good enough to give me time to get a new disk in case of failure but not for daily.

> Somehow you've got active devices set to 3, so just 
>set that back to two. If you look on the wiki it covers the scenario of
>
>converting a three-disk raid to two.
>
>The only thing I will add about a 3-disk raid is that if you want to 
>change it to anything else, you'll have to revert to 2 first, but do 
>that when you want to change things, leave it at 3 for the moment.
>

So I tried with the --grow option to change the number disk in the array but it doesn't want to change...

# mdadm --grow /dev/md0 --raid-devices=2
mdadm: /dev/md0: no change requested
# mdadm --grow /dev/md0 --raid-devices=3
mdadm: Need 1 spare to avoid degraded array, and only have 0.
       Use --force to over-ride this check.

When looking with --examine, sdc3 is marked a replacement device.

# mdadm --examine /dev/sda3 /dev/sdb3 /dev/sdc3|grep 'dev\|Role\|State'
/dev/sda3:
          State : clean
   Device Role : Active device 1
   Array State : AR ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdb3:
          State : clean
   Device Role : Active device 0
   Array State : AR ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdc3:
          State : clean
   Device Role : Replacement device 1
   Array State : AR ('A' == active, '.' == missing, 'R' == replacing)

Thanks,
Aymeric
-- 
Aymeric P.
Phone: +33 685 674 843

Re: mdadm RAID1 with 3 disks active on 2 disks configuration

[Aymeric]

Le 23 février 2020 21:40:44 GMT+01:00, antlists <antlists@youngman.org.uk> a écrit :
>On 23/02/2020 17:38, Aymeric wrote:
>
>Because rebuilding sda is a lot of work. The sensible option would have
>
>been to leave sdb and sdc active, and left sda as spare.

Well, for some reason that didn't happen that way..

>> I used the following command to add sda to the RAID1:
>> # mdadm  /dev/md0 --add /dev/sda3
>>
>>
>> What did I do wrong here? And how to fix it? Do I have to set sdc to
>> faulty and add it back?
>
>I'd just leave it as a 3-disk raid. But no, if you do want to set it 
>back to a 2-disk raid don't do a "faulty and re-add" - that'll just
>make 
>re-sync it again.

I would prefer to revert to sdc as a spare, it's a really a slow disk (5400rpm with a sata1 while others are 7200 at sata3). For a spare disk it's good enough to give me time to get a new disk in case of failure but not for daily.

> Somehow you've got active devices set to 3, so just 
>set that back to two. If you look on the wiki it covers the scenario of
>
>converting a three-disk raid to two.
>
>The only thing I will add about a 3-disk raid is that if you want to 
>change it to anything else, you'll have to revert to 2 first, but do 
>that when you want to change things, leave it at 3 for the moment.
>

So I tried with the --grow option to change the number disk in the array but it doesn't want to change...

# mdadm --grow /dev/md0 --raid-devices=2
mdadm: /dev/md0: no change requested
# mdadm --grow /dev/md0 --raid-devices=3
mdadm: Need 1 spare to avoid degraded array, and only have 0.
       Use --force to over-ride this check.

When looking with --examine, sdc3 is marked a replacement device.

# mdadm --examine /dev/sda3 /dev/sdb3 /dev/sdc3|grep 'dev\|Role\|State'
/dev/sda3:
          State : clean
   Device Role : Active device 1
   Array State : AR ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdb3:
          State : clean
   Device Role : Active device 0
   Array State : AR ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdc3:
          State : clean
   Device Role : Replacement device 1
   Array State : AR ('A' == active, '.' == missing, 'R' == replacing)

Thanks,
Aymeric
-- 
Aymeric P.
Phone: +33 685 674 843
Aymeric