3-way mirror to RAID-6

3-way mirror to RAID-6

[Gandalf Corvotempesta]

Hi to all
I have a server with a 3way mirror.
As I have to increase the total available space, one solution would be to
replace all disks with bigger ones, but this is expensive.

Another solution would be to add just one disk and then reshape from
RAID-1 to RAID-6

Do you have any guide on how to do this reshape on a running array ?
Thank you.

Re: 3-way mirror to RAID-6

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 730 bytes --]

On Mon, Dec 25 2017, Gandalf Corvotempesta wrote:

> Hi to all
> I have a server with a 3way mirror.
> As I have to increase the total available space, one solution would be to
> replace all disks with bigger ones, but this is expensive.
>
> Another solution would be to add just one disk and then reshape from
> RAID-1 to RAID-6
>
> Do you have any guide on how to do this reshape on a running array ?
> Thank you.

I suggest you add the new device with "mdadm --add", then try the
reshape with "mdadm --grow /dev/mdXXX --level=6 --raid-devices-4".

It might not work quite that easily - I think you need to first convert
the raid1 to a 2-way mirror, but that is easy enough

 mdadm --grow /dev/mdxxx --raid-devices=2

NeilBrown

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: 3-way mirror to RAID-6

[Wols Lists]

On 25/12/17 21:53, NeilBrown wrote:
> On Mon, Dec 25 2017, Gandalf Corvotempesta wrote:
> 
>> Hi to all
>> I have a server with a 3way mirror.
>> As I have to increase the total available space, one solution would be to
>> replace all disks with bigger ones, but this is expensive.
>>
>> Another solution would be to add just one disk and then reshape from
>> RAID-1 to RAID-6
>>
>> Do you have any guide on how to do this reshape on a running array ?
>> Thank you.
> 
> I suggest you add the new device with "mdadm --add", then try the
> reshape with "mdadm --grow /dev/mdXXX --level=6 --raid-devices-4".
> 
> It might not work quite that easily - I think you need to first convert
> the raid1 to a 2-way mirror, but that is easy enough
> 
>  mdadm --grow /dev/mdxxx --raid-devices=2

That was my reaction. I'm 99% certain that a three-way will not reshape.
> 
> NeilBrown
> 
Cheers,
Wol

Re: 3-way mirror to RAID-6

[Gandalf Corvotempesta]

2017-12-25 22:53 GMT+01:00 NeilBrown <neilb@suse.com>:
> I suggest you add the new device with "mdadm --add", then try the
> reshape with "mdadm --grow /dev/mdXXX --level=6 --raid-devices-4".

Ok.

> It might not work quite that easily - I think you need to first convert
> the raid1 to a 2-way mirror, but that is easy enough
>
>  mdadm --grow /dev/mdxxx --raid-devices=2

Moving from a 3-way mirror to a 2-way mirror is not an issue.

What do you suggest ? Adding 3 more disks and creating a striped 3-way
mirror or reshaping to a 4-disks RAID-6?

Keep in mind that i'll never run anything with redundancy level < 2
and a simple mirror has only "1" as redundancy level (a striped mirror
is the same: only one disks could be lost in each mirror pair)

Re: 3-way mirror to RAID-6

[Phil Turmel]

On 12/27/2017 05:13 AM, Gandalf Corvotempesta wrote:
> 2017-12-25 22:53 GMT+01:00 NeilBrown <neilb@suse.com>:
>> I suggest you add the new device with "mdadm --add", then try the
>> reshape with "mdadm --grow /dev/mdXXX --level=6 --raid-devices-4".
> 
> Ok.

If that works, you're done.  But the designer of the code thinks it
won't, so don't hold your breath.

>> It might not work quite that easily - I think you need to first convert
>> the raid1 to a 2-way mirror, but that is easy enough
>>
>>  mdadm --grow /dev/mdxxx --raid-devices=2
> 
> Moving from a 3-way mirror to a 2-way mirror is not an issue.

I think you mean "is an issue" ....

> What do you suggest ? Adding 3 more disks and creating a striped 3-way
> mirror or reshaping to a 4-disks RAID-6?
> 
> Keep in mind that i'll never run anything with redundancy level < 2
> and a simple mirror has only "1" as redundancy level (a striped mirror
> is the same: only one disks could be lost in each mirror pair)

.... based on this requirement of yours.

You can't get there from here, at least not while running.  Converting
from raid1 n=2 to any parity-based layout takes advantage of the fact
that the data content of a simple mirror is identical to the data
content of a raid4/5 n=2, since parity of one bit is the same bit.

With some downtime, you could do the following:

1) Unmount and/or deactivate your array (but leave the array running).

2) Reduce your mirror from n=3 to n=2 and *remove* the newly spared
member -- holding it in reserve in case of disaster.

3) Add the new drive as a spare, then --grow to raid6 n=3.  *Wait* for
the reshape to complete (restoring your desired redundancy within the
array).

4) Remount or re-activate your array and resume normal operations.

5) Add the drive from (2) back as a spare (may have to
--zero-superblock) and reshape from raid6 n=3 to raid6 n=4.

There are no-downtime options possible if you have LVM on top of your
array, and if you can add at least a fifth drive to the mix.

Phil

5) Wi

Re: 3-way mirror to RAID-6

[Phil Turmel]

On 12/28/2017 03:33 AM, Gandalf Corvotempesta wrote:
> Il mercoledì 27 dicembre 2017, Phil Turmel <philip@turmel.org
> <mailto:philip@turmel.org>> ha scritto:
> 
>     I think you mean "is an issue" ....
> 
> 
> No, is not an issue as long it's a temporary step...

Ok.  Then:

> I can remove one disk from the 3way mirror for this operation, after
> that, growing from a 2way mirror to a raid6 should be possible, right ?

Yes, I think so.  A quick test with some loop devices on your kernel
will tell you for sure, and if it will take only one step.

> Yes I have LVM and I can add a fifth drive, but I prefer to only add a
> fourth drive and move from current 3way mirror to raid 6

With LVM, it is always possible to set up new drives in an entirely new
raid of the desired configuration, possibly with a "missing" or two, add
it as a physical volume to the same VG, then mirror the LVs across both
arrays.  (Or just pvmove if the target array's redundancy is
sufficient.)  Move redundant drives from old array to new if needed,
then vgreduce to drop the old array.

> It's a mailserver, is a raid6 a performance killer? 

I've never put a mail archive on raid6.  I'd be concerned.  It's
basically a random access workload with small reads and writes.  That is
a recipe that'll maximize the write-amplification slowdown of raid6.

Consider converting to raid10,near3 n=4, or raid10,near3 n=5 to get a
little more capacity but keep the high performance of mirroring.  Raid10
doesn't require the array's member count be a multiple of the number of
copies.

Phil

Re: 3-way mirror to RAID-6

[Gandalf Corvotempesta]

2017-12-28 13:34 GMT+01:00 Phil Turmel <philip@turmel.org>:
> I've never put a mail archive on raid6.  I'd be concerned.  It's
> basically a random access workload with small reads and writes.  That is
> a recipe that'll maximize the write-amplification slowdown of raid6.

Ok, just to stay on the safety side, this is the current 3-way raid1:

# mdadm --detail /dev/md0
/dev/md0:
        Version : 1.2
  Creation Time : Mon Jul 25 12:55:48 2016
     Raid Level : raid1
     Array Size : 488382841 (465.76 GiB 500.10 GB)
  Used Dev Size : 488382841 (465.76 GiB 500.10 GB)
   Raid Devices : 3
  Total Devices : 3
    Persistence : Superblock is persistent

    Update Time : Mon Jan  8 12:26:43 2018
          State : clean
 Active Devices : 3
Working Devices : 3
 Failed Devices : 0
  Spare Devices : 0

           Name : mail:0  (local to host mail)
           UUID : b2a5ed53:42890b73:dc6de22a:1ac12524
         Events : 30658

    Number   Major   Minor   RaidDevice State
       0       8        1        0      active sync   /dev/sda1
       1       8       17        1      active sync   /dev/sdb1
       2       8       33        2      active sync   /dev/sdc1



# pvs
  PV         VG   Fmt  Attr PSize   PFree
  /dev/md0   vg1  lvm2 a-   465,76g    0

# vgs
  VG   #PV #LV #SN Attr   VSize   VFree
  vg1    1   6   0 wz--n- 465,76g    0


# lvs
  LV         VG   Attr   LSize   Origin Snap%  Move Log Copy%  Convert
  lv_dovecot vg1  -wi-ao  10,00g
  lv_log     vg1  -wi-ao  15,00g
  lv_mail    vg1  -wi-ao 410,76g
  lv_root    vg1  -wi-ao  20,00g
  lv_swap    vg1  -wi-ao   8,00g
  lv_tmp     vg1  -wi-ao   2,00g


I want to replace all disks with 2TB SAS disks.

So, which is the best procedure to follow to increse the total usable
space, without loosing redundancy while the system is running ?

I would like to have some confirms from skilled users in this ML.
On my own, i would replace one disk per time (fail+remove+add) and
wait for resync to complete, then, resize the array:

mdadm --grow /dev/md0 --bitmap none
mdadm --grow /dev/md0 --size=max

wait for it to finish

then: pvresize /dev/md0, lvresize (each LVs), resize2fs (each FS)

Is this ok ?

Re: 3-way mirror to RAID-6

[Wols Lists]

On 08/01/18 11:34, Gandalf Corvotempesta wrote:
> So, which is the best procedure to follow to increse the total usable
> space, without loosing redundancy while the system is running ?
> 
> I would like to have some confirms from skilled users in this ML.
> On my own, i would replace one disk per time (fail+remove+add) and
> wait for resync to complete, then, resize the array:

Well, that's certainly not best practice!

If you can, you replace one disk at a time with --replace. That will
copy the drive in-place with *no* reduction in redundancy. Doing a
fail/remove removes one level of redundancy.

Okay, you need spare ports to do a --replace so it's not always
possible, but it's always desirable. And with enough spare ports, you
can replace more than one drive at once, again with NO loss of redundancy.

Cheers,
Wol

Re: 3-way mirror to RAID-6

[Gandalf Corvotempesta]

2018-01-08 16:20 GMT+01:00 Wols Lists <antlists@youngman.org.uk>:
> Okay, you need spare ports to do a --replace so it's not always
> possible, but it's always desirable. And with enough spare ports, you
> can replace more than one drive at once, again with NO loss of redundancy.

I have 3 ports available, so I can replace 3 disks at once.
Any hint on the exact command so that I can safely test on a test server ?

Re: 3-way mirror to RAID-6

[Wol's lists]

On 08/01/18 15:25, Gandalf Corvotempesta wrote:
> 2018-01-08 16:20 GMT+01:00 Wols Lists <antlists@youngman.org.uk>:
>> Okay, you need spare ports to do a --replace so it's not always
>> possible, but it's always desirable. And with enough spare ports, you
>> can replace more than one drive at once, again with NO loss of redundancy.
> 
> I have 3 ports available, so I can replace 3 disks at once.
> Any hint on the exact command so that I can safely test on a test server ?
> 
Search the man page for --replace, but (off the top of my head) I think 
it's as simple as

mdadm /dev/md0 --add /dev/sdg --replace /dev/sdc

or

mdadm /dev/md0 --add /dev/sdg /dev/sdh /dev/sdi /dev/sdj --replace 
/dev/sdc /dev/sdd /dev/sde /dev/sdf

The man page says that the --replace option tells the raid code to 
failover the selected device as soon as a spare becomes available, so 
using --add --replace gives it the spare and tells it to use it.

With this number of devices to play with, I'd make sure to make a note 
of the original UUIDs and use those where possible - if linux plays with 
your sdx or you muddle your cables or whatever, then you could easily 
make an array-damaging typo ...

Cheers,
Wol